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I 


INTRODUCTION 


A.  PURPOSE 

The  purpose  of  this  thesis  is  to  study  the  extent  to  which  a  series  of  factors 
influenced  development  of  the  Target  Acquisition  Designation  System/Pilot  Night  Vision 
System  (TADS/PNVS).  The  research  findings  and  conclusions  will  be  primarily  based 
upon  answers  to  a  questionnaire  completed  by  the  Government  and  Contractor  Program 
Managers  (PM)  or  Deputy  Program  Managers  (DPM)  and  their  staffs,  supplemented  by 
interviews  with  these  individuals. 
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Figure  1.  AH-64A  Mission  Equipment  Package  Architecture  from  TADS/PNVS  Interfaces 
with  other  Mission  Equipment  of  the  AH-64  Helicopter,  Martin  Marietta  Aerospace 

International,  May  1983. 
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B.  BACKGROUND 

The  U.S.  Army  developed  a  variety  of  systems  in  the  1970s  and  1980s,  based  on 
experience  gained  in  the  Viet  Nam  War.  Many  of  these  systems  did  not  see  significant 
actual  combat  usage  until  1991,  during  Operation  Desert  Storm  in  Iraq. 

The  first  shots  of  Operation  Desert  Storm  were  fired  by  AH-64A  Apache 
Helicopters  (Task  Force  Normandy)  on  “January  17,  1991”.  The  TADS/PNVS  was  used 
to  acquire  the  targets.  At  first,  they  used  the  heat  from  the  target  to  guide  the  missiles. 
When  a  flash  was  distracting  some  missiles,  they  switched  to  optical  guidance  (From  Hot 
Air  to  Hellfire  -  James  W.  Bradin,  ©  1994). 

The  targets,  two  state-of-the-art  Soviet-built  radar  sites,  which  threatened  to  give 
early  warning  of  the  initiation  of  the  air  campaign,  were  simultaneously  attacked  at  2:38 
am.  The  targets  were  completely  destroyed.  This  allowed  the  allies  to  fly  surreptitiously 
right  in  and  bomb  Iraq  (Bradin,  1994). 

Originally,  the  Target  Acquisition  Designation  System  /  Pilot  Night  Vision 
System  (TADS/PNVS)  was  conceived  by  The  U.S.  Army  Missile  Command  (MICOM), 
which  initially  led  the  developmental  effort.  It  was  subsequently  transitioned  to  the 
Apache  Attack  Helicopter  Program  Manager’s  Office  (AAH  PMO).  The  U.S.  Army 
developed  TADS  as  a  sensor  for  the  Hellfire  missile  system.  TADS/PNVS  was 
developed  in  the  1970s  and  1980s  under  control  of  the  TADS  Program  office,  which  was 
a  part  of  the  AAH  Program  Management  Office. 

C.  RESEARCH  QUESTIONS 

1.  Primary  Question: 

What  was  the  simulation  and  testing  strategy  for  the  system,  and  did  that  strategy 
adequately  evaluate  the  system  for  its  ultimate  operational  use? 

2.  Secondary  Questions: 

a.  To  what  extent  did  the  maturity  (at  project  initiation)  of  the  critical 
technologies  being  integrated  into  the  TADS/PNVS  system  influence  the  development? 
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b.  How  were  the  organizations  that  had  developed  these  critical 
technologies  involved  during  system  development? 

c.  To  what  extent  was  there  user  support  and  funding  stability  during 
system  development? 

d.  How  effectively  were  (what  we  now  call)  integrated  product  teams 
employed  during  development? 

e.  What  was  the  key  issue  that  the  PM  had  to  deal  with  during  program 
development  and  how  was  it  dealt  with? 

D.  SCOPE  OF  THE  THESIS 

The  thesis  will  focus  on  TADS/PNVS  development,  will  note  how  well  it  met  its 
cost,  schedule,  and  perfonnance  goals,  and  will  also  touch  briefly  upon  its  successful  use 
in  DESERT  STORM.  It  will  consider  the  critical  technologies  of  TADS  /  PNVS  and 
whether  they  were  effectively  implemented  in  this  system.  The  research  method  will  be  a 
case  study,  developed  by  use  of  questionnaires  and  interviews. 

This  thesis  explores  the  interrelationship  of  players  such  as  users,  Government 
PMO,  contractors,  technology  developers,  and  testers  in  carrying  out  the  development, 
production,  and  fielding  of  the  system.  In  addition,  factors  such  as  the  effective  use  of 
integrated  product  teams,  the  maturity  and  production  readiness  of  the  critical 
technologies,  the  role  played  by  testing  and  simulation,  the  relationship  between  testing 
and  operational  use,  and  the  key  issue  faced  during  the  program  and  its  resolution  are 
examined.  Project  outcomes,  in  cost,  schedule  and  perfonnance  in  terms  of  Desert 
Storm,  are  identified. 

E.  METHODOLOGY 

Research  approach  consists  of  determining  three  TADS/PNVS  Critical 
Technologies  in  consultation  with  the  TADS/PNVS  technology  community;  sending 
questionnaires  out  to  current  and  former  Government  and  Contractor  Program  Managers 
and  their  staffs;  then  conducting  interviews  with  them;  and  analyzing  this  data.  I  used  a 
tape-recorder  attached  to  the  telephone,  when  the  interviewee  consented  to  being 
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recorded.  Additionally,  I  analyzed  the  testing  of  these  systems  to  detennine  if  it  were 
adequate  to  prove  the  system’s  combat  readiness.  The  results  of  the  initial  interviews 
were  written  up  and  I  determined  where  I  had  coverage  or  gaps  in  my  data.  I  then 
conducted  follow-up  interviews,  or  questioned  alternate  personnel,  who  fdled  in  data 
gaps.  Next  I  used  my  overall  understanding  of  the  development  to  integrate  the  results 
from  all  survey  questionnaires  into  one  composite  response  survey  result,  to  be  used  for 
the  subsequent  crosscutting  analysis. 


F.  ORGANIZATION 

Chapters  II  through  VI  discuss  the  secondary  research  questions,  and  Chapter  VII 
addresses  the  primary  research  question.  In  each  chapter,  I  summarize  the  data  collected 
from  the  survey.  I  will  introduce  the  data,  and  also  mention  briefly  the  way  in  which  it 
was  acquired. 

I  analyzed  the  data,  comparing  responses  to  various  questions,  and  between  the 
Government  and  contractor  respondents,  as  well  as  advantages  and  disadvantages, 
analyzing  them  in  terms  of  the  primary  and  secondary  questions.  Then  I  will  discuss 
lessons  learned,  and  draw  conclusions  and  make  recommendations. 

Chapter  II:  Mature  Technologies:  To  what  extent  did  the  maturity  (at  project 
initiation)  of  the  critical  technologies  being  integrated  into  the  TADS/PNVS  system 
influence  the  development? 

Chapter  III:  Development  Organizations:  How  were  the  organizations  that  had 
developed  these  critical  technologies  involved  during  system  development? 

Chapter  IV:  User  Support  and  Funding  Stability:  To  what  extent  was  there 
user  support  and  funding  stability  during  system  development? 

Chapter  V:  Integrated  Product  Teams:  How  effectively  were  (what  we  now 
call)  integrated  product  teams  employed  during  development? 

Chapter  VI:  Key  Program  Manager  Issue:  What  was  the  key  issue  that  the  PM 
had  to  deal  with  during  program  development  and  how  was  it  dealt  with? 
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Chapter  VII:  Primary  Question  -  Simulation  and  Testing  Strategy:  What  was 
the  simulation  and  testing  strategy  for  the  system,  and  did  that  strategy  adequately 
evaluate  the  system  for  its  ultimate  operational  use? 

In  the  back,  there  is  a  list  of  Acronyms  and  Definitions. 

In  Appendix  A,  I  will  provide  the  Composite  Questionnaire  Response.  This  will 
be  a  composite  of  the  Government  and  developer  /  contractor  responses. 

G.  BENEFITS  OF  THE  STUDY 

This  research  will  study  the  issues  and  relationships  associated  with  the 
development  of  the  TADS/PNVS.  This  case  study  is  one  of  a  series  being  prepared  under 
an  ongoing  research  effort  sponsored  by  Headquarters  U.S.  Anny  Material  Command 
(AMC).  The  U.S.  Army  Aviation  and  Missile  Command  (AMCOM)  has  contracted  with 
the  University  of  Alabama  in  Huntsville  to  do  this  research,  utilizing  students.  After  the 
Case  Study  research  is  completed,  the  Principal  Investigators  at  UAH  and  Massachusetts 
Institute  of  Technology  (MIT)  will  do  a  crosscutting  analysis  to  identify  key  factors 
common  to  all  the  systems  studied  that  can  be  used  to  guide  future  decision-making.  The 
case  studies  will  be  made  available  to  the  Defense  System  Management  College  (DSMC) 
and  the  Naval  Postgraduate  School  (NPS)  to  use  in  both  teaching  and  research. 
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II.  MATURE  TECHNOLOGIES 


A.  RESEARCH  QUESTION 

This  chapter  answers  the  research  question,  “To  what  extent  did  the  maturity  (at 
project  initiation)  of  the  critical  technologies  being  integrated  into  the  TADS/PNVS 
system  influence  the  development?”  This  chapter  will  look  at  the  maturity  of  the  critical 
technologies  in  terms  of  (1)  project  outcomes,  (2)  technology  readiness,  and  (3)  project 
timeline.  I  will  introduce  the  data,  then  analyze  it,  and  finally  draw  conclusions. 


1 .  SURVIVAL  EQUI PMENT  STORAGE  BAY 

2.  AET  AVIONICS  BAY  ACCESS  DOOR 

3.  WIRE  STRIKE  PROTECTION  SYSTEM  (WSPS) 

4.  CPGDOOR 

5.  PILOT  DOOR 

6.  SEARCHLIGHT 

7.  AMMUNITION  KAY  ACCESS  DOOR 

8.  TAIL  LANDING  GEAR 


9.  CANOPY  JEniSON  HANDLE  ACCESS  DOOR2 
10  CHAFF  DISPENSER 

11.  MAS  I  MOUNTED  ASSEMBLY  (MMA)  [1 

12.  LEFT  SIDE  EXTENDED  FORWARD  AVIONICS  BAY  (EFAB) 

13.  I R  JAMMER 

14.  LASER  WARNING  SENSORS 

15.  AIR  DATA  SYSTEM  IADS1  SENSOR 

m  RIGHT  SIDE  EXTENDED  FORWARD  AVIONICS  BAY  (EFAB) 


LBA-0001 


Figure  2.  Longbow  Apache  General  Arrangement,  from  TM  1-1520-251-10,  Technical 
Manual,  Operator's  Manual  for  Helicopter,  Attack,  AH-64D  Longbow  Apache,  15  Dec 

1998 
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B.  DATA 

Data  was  acquired  using  questionnaires  /  survey  and  interviews.  From  all  this 
data,  a  combined  survey  was  created  (see  Appendix  A.)  I  extracted  the  data  in  this  and 
subsequent  chapters  from  the  combined  survey. 

In  the  survey,  each  question  has  a  letter  and  number  (i.e.  Tl)  except  for  questions 
on  page  1.  In  the  survey,  the  format  is:  question,  then  multiple-choice  answers  for  some 
questions,  and  possibly  a  blank  for  answers.  When  I  extract  information  from  the  data,  it 
will  always  reference  this  numbering  system,  and  give  the  question  and  the  answer  that 
was  chosen,  not  all  possible  choices  (i.e.  X  4.  All  of  the  above.)  Responses  to  questions 
are  in  italics. 

To  better  understand  this  data,  I  include  the  critical  technologies: 

Tl.  Now  identify  one  or  more  (up  to  3)  technologies  that  were  incorporated 
into  the  system  you  are  studying.  These  technologies  should  be  among  those  central  to 
the  success  of  the  system  (critical). 


Technology  A 

Line-of-Sight  Stabilization 

Technology  B 

FLIR  target  acquisition 

Technology  C 

Laser  to  sensor  bore  sight 

Table  1.  Critical  Technologies 


1.  Outcomes? 

This  contains  survey  questions  01  through  09,  Project  Outcomes. 


Project  Outcomes 

01.  Project  Acceptance.  Was  the  SYSTEM  accepted  to  be  put  into  Production? 
This  is  initial  acceptance,  not  whether  it  actually  ended  up  in  production. 

X  3.  Yes,  the  System  was  accepted  for  production 
02.  After  the  SYSTEM  was  accepted  and  was  in  Transition  to  Production,  how 
many  additional  changes  in  the  designs  and  processes  were  later  required  before  the 
System  was  taken  into  full  production?  X  1.  Many  serious  changes 

There  was  a  large  amount  of  work.  TADS  pointing  angle  accuracy  was  a 
big  problem.  They  had  to  work  on  getting  a  noise-free  FLIR.  And  they  needed  to  work 
on  consistency  of  Line-of-Sight  (LOS)  Stabilization  -  they  made  repeated  changes  to  meet 
this  specification  requirement.  The  delivery  rate  of  10,  and  then  12  per  month 
exacerbated  the  problem. 
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03.  Did  the  SYSTEM  go  into  full  production? 

X  3.  Yes,  the  System  was  put  into  full  production. 

04.  For  each  of  the  technologies  A,  B,  and  C  above,  to  what  extent  was  each  used 
in  the  System  as  it  was  produced? 

Technology  ABC 

4.  Yes,  the  technology  was  used  as  planned.  4.  X  4.  X  4.  X 

After  the  early  stages  of  development,  LOS  stabilization  never  really  became  an 
issue  any  more.  FLIR  acquisition  ranges  were  met  in  the  later  stages  of  development  and 
were  not  a  problem  in  the  production  hardware.  Bore-sight  performance  continued  to  be 
an  issue  into  the  early  stages  of production.  The  cost  of  the  fixes  were  not  major  but  took 
a  lot  of  time  to  work  out.  All  three  technologies  were  essential  to  the  performance  of  the 
TADS  and  thus  had  to  be  successfully  used  in  the  final  system. 

05.  After  the  SYSTEM  reached  Transition  to  Production,  did  the  project  go  to 
Production  as  quickly  as  it  should  have?  X  2.  One  to  six  months 

06.  After  the  SYSTEM  was  actually  in  Production,  how  many  additional 
changes  in  designs  and  processes  were  required?  X  3.  Minor  changes 

“Again,  the  contractor  was  incentivised  to  make  reliability  improvement  changes 
and  under  the  warranty  program  could  make  changes  to  improve  reliability  and  thereby 
save  the  contractor  (and  ultimately  the  Government)  money.  Producibility  changes  were 
also  made  mostly  because  of  parts  that  were  no  longer  available.  ” 

08.  Did  the  System  Development  program,  as  implemented,  come  in  on  budget? 

X  3.  The  project  significantly  exceeded  budget. 

“As  stated  above  there  were  significant  overruns  to  the  development  contracts. 
The  “Maturity  phase  contract  with  Martin  Marietta  started  off  at  about  $45M  and  ended 
up  at  about  twice  that.  However,  TADS/PNVS  was  not  a  separate  line  item  in  the  budget 
but  was  just  part  of  the  AH-64  budget  and  this  overrun  was  covered  within  the  AH-64 
budget.  ’’ 

09.  Did  the  System  as  it  was  implemented  meet  the  project’s  technical  goals  and 
functional  requirements?  X  1.  The  results  met  or  exceeded  technical  goals. 

09.  (Note:  there  are  two  questions  designated  09.)  Did  the  System  have 
problems  in  the  field  under  operational  conditions  in  Desert  Storm? 

X  3.  No,  the  system  was  deployed  and  encountered  no  noticeable  loss  of 

effectiveness. 

2.  Technology  Readiness? 

This  contains  survey  questions  T5,  T6  T7,  and  Page  1  (SP,  D,  and  TP  questions.) 
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Check  (V)  the  best  answer  for  each 
technology. 

Technologv 

A 

Technologv 

B 

Technologv 

C 

T5.  When  System  planning  and  pre¬ 
development  began,  technology  TRL  was: 

4 

4 

3 

T6.  When  System  went  into  Development, 
technology  TRL  was: 

5 

5 

4 

T7.  When  System  reached  Transition  to 
Production,  technology  TRL  was: 

9 

9 

8 

Table  2.  Data  for  Questions  T5,  T6,  and  T7 


TRL  =  Technology  Readiness  Level  (TRL  numbers  are  defined  at  the  end  of 
Appendix  A,  Combined  Survey) 

When  SP  (System  Planning  phase)  started,  stabilization  technology  was  not  new 
and  had  had  many  applications,  but  none  to  this  difficult  an  application  in  a  helicopter 
flight  environment.  Likewise,  due  to  the  work  on  FLIR  technology  by  NVL  [U.S.  Army 
Night  Vision  Labs]  there  was  a  significant  technology  base  to  draw  on;  however,  meeting 
target  detection  and  recognition  requirements  was  a  very  difficult  goal  and  integration  of 
a  FLIR  meeting  these  requirements  into  the  stabilized  turret  was  a  real  challenge.  The 
bore-sight  problem  was  recognized  as  critical  from  the  very  beginning,  but  achieving  a 
bore-sight,  which  met  accuracy  requirements  and  remained  stable  over  environmental 
extremes  proved  very  difficult  and  tenuous.  Since  bore-sight  stability  is  impacted  by 
many  factors  in  all  sensors,  bore-sighting  components  and  the  stabilized  turret,  it  was  not 
possible  to  address  bore-sight  shortcomings  until  the  entire  TADS  system  was  designed, 
built,  and  tested  for  other  areas  of performance. 


SP.  In  what  organization  was  the  primary  work  leading  up  to  this  point 
accomplished?  There  were  really  three  important  organizations  that  contributed.  The 
ASH  PM  (Advanced  Scout  Helicopter)  prior  to  being  cancelled  and  the  AAH  PM  were 
the  driving  force  for  establishing  and  planning  the  program.  The  technology  work  was 
being  led  by  the  MICOM  G&C  (Guidance  and  Control)  lab  and  The  Night  Vision  Lab 
(NVL).  Contractor  S&T  (Science  and  Technology)  organizations  were  doing  their  own 
work  in  response  to  the  anticipated  requirement  for  the  ASH  and  AAH  programs. 

“The  MICOM  G&C  lab  was  the  developer  of  the  Hellfire  missile  (concept)  for 
which  the  TADS  acquires  and  designates  targets.  Major  systems  requirements  such  as 
Total  Pointing  Error  (TPE)  for  the  laser  designator  were  defined  by  the  G&C  lab  based 
on  testing  and  simulations.  They  also  did  early  work  on  the  laser  hardware  that  does  the 
designation.  NVL  was  the  developer  of  FLIR  technology,  which  was  used  in  the  TADS 
night  sight  and  the  PNVS.  They  were  responsible  for  the  development  and  eventually 
production  of  the  FLIR  common  modules,  which  are  used  in  the  TADS  Night  Sight. 
Significant  support  was  given  by  these  labs  and  Frankfort  Arsenal  (fire  control,  optics)  in 
formulating  requirements,  evaluating  proposals,  and  monitoring  development  progress.  ” 

D.  What  was  the  Technology  Readiness  Level  (refer  to  page  8)  for  the  SYSTEM 
on  this  date?  Level  3:  The  system  met  the  spec,  but  not  consistently.  They  had  proved 
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that  the  gimbals  would  meet  the  LOS  (Line-of-Sight)  Stabilization.  A  prototype  was  built 
in  the  proposal  phase. 

What  was  the  nature  of  the  Anny  Lab/Center’s  involvement?  (Engineering 
support?  Simulation  or  testing?  Integration?  Requirements  interpretation?)  G&C  Lab, 
NVL,  and  Frankfort  Arsenal  provided  engineering  support,  simulation,  and  requirements 
interpretation. 

TP  (Transition  to  Production).  In  what  organization  was  the  primary  work  in  the 
period  from  D  to  TP  accomplished?  Martin  Marietta,  Orlando,  the  Prime  Contractor. 
This  work  was  done  under  a  project  management  (PM)  organization. 

What  was  the  nature  of  the  Anny  Lab/Center’s  involvement?  (Engineering 
support?  Simulation  or  testing?  Integration?  Requirements  interpretation?)  MICOM 
G&C,  NVL,  and  Frankfort  Arsenal  continued  to  provide  engineering  support,  simulation, 
test  witnessing,  and  requirements  interpretation. 

3.  Timeline? 

This  covers  survey  questions  on  Page  1.  The  timeline  data  from  page  1  is  as 
follows: 

SP.  What  was  the  approximate  starting  date  of  systems  planning  and  pre¬ 
development  work?  This  date  is  when  planning  work  began  on  the  integrated  system. 
The  systems  concept  and  applications  had  been  formulated,  but  applications  were  still 
speculative.  There  was  no  proof  or  detailed  analysis  to  support  the  approach.  SYSTEMS 
PLANNING  START  DATE  (SP):  _ H976  (mo/yr)  [TRL2  at  system  level] 

D.  Date  when  Development  started.  Typically  at  this  date,  funding  started  for 
system  advanced  or  engineering  development,  a  Government  project  office  was  formed 

and  Prime  Contractor(s)  selected.  DEVELOPMENT  START  DATE  (D):  _ 11977 

(mo/yr) 

TP.  Date  of  achieving  “Transition  to  Production”  when  producible  system 
prototype  has  been  demonstrated  in  an  operational  environment.  Prototype  is  near 
or  at  planned  operational  system,  produced  on  small  scale.  TRANSITION  TO 
PRODUCTION  (TP)  DATE: _ 11980  (mo/yr)  (TRL7  at  system  level) 

Additional  Timeline  data  is  found  in  From  Hot  Air  to  Hellfire  -  James  W.  Bradin, 
©  1994. 


Date 

Event 

10  Dec  1976: 

Down  select  to  Hughes  YAH-64A 

FY  1982: 

Congress  approves  LRIP,  $444.5  M  Contract  for  1 1  aircraft 

Table  3.  From  Hot  Air  to  Hellfire  Timeline 
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Further  timeline  data  was  found  in  “Selected  Acquisition  Report  (SAR),  30 
September  1992”,  an  annual  report  on  the  status  of  the  AH-64  Apache  Helicopter 


development  program. 


Date 

Event 

22  June  1973 

Competitive  Phase  I,  Development  Contracts  awarded  to  Hughes 
Helicopters  and  Bell  Helicopters  Textron,  Inc 

7  Dec  1976 

DSARC  approved  AAH  entry  into  full  scale  development  (Phase  II)  and 
Secretary  of  the  Army  selected  Hughes  Helicopters,  Model  YAH-64 

10  Mar  1977 

TADS/PNVS  directed  for  development,  contracts  awarded  to  Martin 
Marietta  and  Northrop  Corporation. 

30  Jan  1981 

Army  awarded  Long  Lead  Time  contract  to  MMOA  (TADS/PNVS) 

20  Feb  1981 

Army  LLTI  contract  to  Hughes  (AH-64) 

Jun-Aug 

1981 

Operational  Test  (OT  II)  was  completed  on  time  at  Ft.  Hunter-Ligett 

18  Nov  1981 

Army  System  Acquisition  Review  Council  (ASARC)  III  was  completed 

26  Mar  1982 

DSARC  III  held,  initial  production  of  Apaches  approved 

April  1982 

Production  contracts  awarded  to  Hughes,  MMOA,  and  General  Electric 
(engines) 

Early  1984 

McDonnell  Douglas  acquired  Hughes  Helicopter 

26  Jan  1984 

McDonnell  Douglas  Helicopter  Company  (MDHC)  first  production 
aircraft  (PV01)  rolled  out 

22  July  1986 

Initial  Operational  Capability 

1 

’able  4.  Selected  Acquisition  Report  (SAR)  Timeline 

Test  Plan  for  TADS/PNVS  Competitive  Development 


1  Dec  1979 
to 

29  Feb  1980 


The  TADS/PNVS  competitive  development  test  was  conducted  at  Yuma 
Proving  Grounds  (YPG).  It  was  a  fly-off  between  the  Martin  Marietta 
Corporation  and  Northrop  Corporation  TADS/PNVS  advanced 
prototypes,  each  mounted  on  AH-64  aircraft. 


Table  5.  TADS/PNVS  Competitive  Development  Timeline 


C.  ANALYSIS 

This  section  will  analyze  the  maturity  of  the  critical  technologies  in  terms  of  (1) 
Outcomes,  (2)  Technology  Readiness,  and  (3)  Timeline. 
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1. 


Outcomes? 


The  three  critical  technologies  of  the  TADS/PNVS  are:  Laser  to  sensor  bore-sight 
(LSBS),  Line-of-Sight  Stabilization  (LOSS),  and  Forward-looking  Infra  Red  (FLIR) 
Target  Acquisition  (FTA).  The  critical  technologies  were  used  as  originally  planned.  All 
three  of  the  technologies  were  essential;  TADS  would  not  have  worked  without  them. 
LOS  Stabilization  was  fixed  in  early  development,  FTA  in  later  development.  Bore 
sighting  wasn’t  finalized  until  early  production  -  a  lot  of  time  was  needed,  but  not  all  that 
much  money.  Although  the  technologies  were  immature  at  the  beginning  of 
development,  the  developer  persevered  and  the  system  was  eventually  accepted  for  full 
production. 

Developing  a  system  such  as  TADS/PNVS  is  a  tremendous  amount  of  work.  This 
work  paid  off  and  the  system  was  very  mature  when  it  transitioned  to  production.  A  lot 
of  work  had  to  be  done  to  both  get  ready  for  production,  and  in  the  transition  to 
production  and  the  early  stages  of  production.  Changes  included  work  on  pointing  angle 
accuracy,  noise-free  FLIR,  and  consistency  of  Line-of-Sight  (LOS)  Stabilization,  which 
required  repeated  changes.  The  required  delivery  rates  of  10,  and  then  12  per  month, 
ramping  up  from  1  per  month,  increased  the  level  of  difficulty. 

In  going  to  production,  the  system  only  experienced  a  short  delay  of  one  to  six 
months.  There  were  some  minor  changes  during  TP,  in  order  for  the  system  to  meet  or 
improve  performance.  Similarly,  there  were  some  minor  changes  to  the  system  while  in 
production,  mostly  to  increase  system  reliability.  The  contractor  had  a  financial  incentive 
to  improve  reliability  (which  eventually  saves  the  Government  money  also.)  They  also 
made  producibility  changes  due  to  parts  becoming  obsolete. 

There  was  a  significant  increase  in  development  costs.  The  original  TADS/PNVS 
contract  was  for  $45  million,  and  it  ended  up  costing  twice  that  amount.  However,  the 
system  met  or  exceeded  technical  performance  goals.  The  system  was  deployed  on  the 
AH-64A  Apache  Helicopter,  and  performed  effectively  in  Operation  Desert  Storm. 
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2.  Technology  Readiness? 

When  system  planning  and  pre-development  began,  two  of  the  three  critical 
technologies  (Line-of-Sight  Stabilization  and  FLIR  target  acquisition)  had  been  verified 
in  breadboard  form  in  a  laboratory  environment  (Technology  Readiness  Level  4).  LOSS 
had  never  been  tried  on  a  helicopter  -  a  high-vibration  environment.  The  third 
technology  (Laser  to  sensor  bore-sight  (LSBS))  had  only  been  verified  by  a  combination 
of  laboratory  work  and  analytical  studies  (TRL  3).  Three  groups  did  most  of  the 
technical  work:  the  MICOM  Guidance  and  Control  (G&C)  lab,  the  US  Army  Night 
Vision  Lab  (NVL),  and  contractor  science  and  technology  group.  Additionally,  Frankfort 
Arsenal  gave  support  in  fire  control  and  optics. 

By  the  time  the  system  was  in  development,  Laser  to  sensor  bore-sight  had  been 
verified  completely  in  a  laboratory  environment,  and  the  other  two  technologies  had  been 
verified  in  a  realistic,  though  simulated  environment  (TRL  5).  This  TRL  indicates  the 
technologies  were  advanced  enough  for  the  development  phase  to  start,  but  not  yet  ready 
for  fielding.  They  had  built  a  prototype,  and  the  system  met  the  specification,  though  not 
consistently.  The  U.S.  Anny’s  contribution  came  from  the  G&C  Lab,  NVL,  and 
Frankfort  Arsenal,  which  provided  engineering  support,  simulation,  and  requirements 
interpretation. 

When  the  system  reached  the  transition  to  production,  the  technologies  were 
considerably  more  advanced.  An  actual  system  had  been  tested,  and  the  Laser  to  sensor 
bore-sight  had  been  qualified  in  test  and  demonstration.  The  technology  was  proven  in 
its  final  form.  The  other  two  technologies,  in  final  form,  had  also  been  successfully 
tested  in  a  realistic  operational  environment.  At  this  point  the  system  was  given  the  go- 
ahead  for  production. 

There  were  still  some  production  reliability  and  manufacturability  issues  to  work 
out,  but  the  essential  system  was  ready.  Bore-sight  stability  is  affected  by  a  number  of 
characteristics  of  all  sensors,  bore-sighting  components,  and  the  stabilized  turret.  This 
makes  doing  the  bore-sight  design  difficult  until  after  the  rest  of  the  TADS  system  has 
been  designed,  built  and  tested.  During  this  phase,  Martin  Marietta  did  the  primary  work. 
The  PMO  oversaw  this  effort,  and  G&C,  NVL,  and  Frankfort  Arsenal  provided  support. 
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3. 


Timeline? 


The  following  table  is  a  timeline  of  the  TADS/PNVS  Program  from  Systems 
Planning  (SP),  though  Development  (D),  to  the  Transition  to  Production  (TP).  As  you 
can  see  from  the  table,  the  Technology  Readiness  Level  (TRL)  of  the  three  critical 
technologies  gradually  increased  as  the  program  progressed.  [See  definitions  of  TRL  at 
end  of  Appendix  A,  Combined  Survey.] 


Key  Program  Start  Dates 

Year 

Technology 
A  &B  TRL 

Technology 
C  TRL 

Systems  Planning  (SP) 

1976 

4 

3 

Development  Start  (D) 

1977 

5 

4 

Transition  to  Production  (TP) 

1980 

9 

8 

Table  6.  Program  Timeline 


The  following  timeline  is  compiled  by  merging  these  dates  in  with  data  from 
other  TADS  documents  (From  Hot  Air  to  Hcllfi re,  Selected  Acquisition  Report  (SAR), 
and  Test  Plan  for  TADS/PNVS  Competitive  Development). 
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Date 

Event 

Ref. 

22  June  1973 

Competitive  Phase  I,  Development  Contracts  awarded  to  Hughes 
Helicopters  and  Bell  Helicopters  Textron,  Inc 

SAR 

1976 

Systems  Planning  (SP) 

Survey 

7  Dec  1976 

DSARC  approved  AAH  entry  into  full  scale  development  (Phase  II) 
and  Secretary  of  the  Army  selected  Hughes  Helicopters,  Model 
YAH-64 

SAR 

10  Dec  1976: 

Down  select  to  Hughes  YAH-64  A 

Bradin 

10  Mar  1977 

TADS/PNVS  directed  for  development,  contracts  awarded  to  Martin 
Marietta  and  Northrop  Corporation. 

SAR 

1977 

Development  Start  (D) 

Survey 

1  Dec  1979 
to 

29  Feb  1980 

The  TADS/PNVS  competitive  development  test  was  conducted  at 
Yuma  Proving  Grounds  (YPG).  It  was  a  fly-off  between  the  Martin 
Marietta  Corporation  and  Northrop  Corporation  TADS/PNVS 
advanced  prototypes,  each  mounted  on  AH-64  aircraft. 

CD 

Test 

Plan 

1980 

Transition  to  Production  (TP) 

Survey 

30  Jan  1981 

Army  awarded  Long  Lead  Time  contract  to  MMOA  (TADS/PNVS) 

SAR 

20  Feb  1981 

Army  LLTI  contract  to  Hughes  (AH-64) 

SAR 

Jun-Aug 

1981 

Operational  Test  (OT  II)  was  completed  on  time  at  Ft.  Hunter-Ligett 

SAR 

18  Nov  1981 

Army  System  Acquisition  Review  Council  (ASARC)  III  was 
completed 

SAR 

FY  1982 

Congress  approves  LRIP,  $444.5  M  Contract  for  1 1  aircraft 

Bradin 

26  Mar  1982 

DSARC  III  held,  initial  production  of  Apaches  approved 

SAR 

April  1982 

Production  contracts  awarded  to  Hughes,  MMOA,  and  General 
Electric  (engines) 

SAR 

Early  1984 

McDonnell  Douglas  acquired  Hughes  Helicopter 

SAR 

26  Jan  1984 

McDonnell  Douglas  Helicopter  Company  (MDHC)  first  production 
aircraft  (PV01)  rolled  out 

SAR 

22  July  1986 

Initial  Operational  Capability 

SAR 

Table  7.  Overall  Program  Timeline 


D.  CONCLUSIONS 

This  section  will  draw  conclusions  concerning  the  maturity  of  the  critical 
technologies.  Conclusions  concerning  the  timeline  will  be  interspersed  with  other 
sections. 


1.  Outcomes? 

The  critical  technologies  were  all  essential  to  the  TADS/PNVS  program:  Laser  to 


sensor  bore-sight  (LSBS),  Line-of-Sight  Stabilization  (LOSS),  and  Forward-looking  Infra 
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Red  (FLIR)  Target  Acquisition  (FTA).  The  TADS/PNVS  was  significantly  beyond 
existing  technology,  and  there  was  a  good  deal  or  risk  to  overcome.  The  amount  of  work 
required  was  greater  than  planned,  but  the  developer  and  the  PMO  were  able  to  complete 
the  development  and  deliver  a  functioning  system. 

This  additional  work  effort  carried  over  into  the  transition  to  production,  and  in 
early  production,  but  the  reward  was  that  they  were  at  a  fairly  good  level  of  readiness. 
Operational  Testing  was  completed  on  time  at  Fort  Hunter-Ligett  in  June -August  1981. 
Changes  to  the  system  were  critical,  to  meet  or  improve  performance,  to  increase  system 
reliability  and  to  improve  reliability,  but  did  not  delay  system  production  very  much.  The 
TADS/PNVS  contract  cost  twice  the  amount  originally  contracted,  from  $45  million  to 
about  $90  million;  but  the  system  met  its  technical  objectives  and  performed  well  in 
Operation  Desert  Stonn. 

2.  Technology  Readiness? 

At  the  start  of  system  planning  (1976)  and  pre-development,  the  TADS/PNVS 
critical  technologies  were  verified  at  Technology  Readiness  Levels  3  or  4  (in  a  lab 
environment  or  by  analytical  studies.)  By  the  time  the  system  was  in  development 
(1977),  this  had  advanced  to  TRL4  or  TRL5  (verified  in  realistic,  though  simulated 
environment.)  When  the  system  reached  the  transition  to  production  (1980),  the  critical 
technologies  were  at  TRL8  (qualified  in  test  and  demonstration)  or  TRL9  (an  actual 
system  had  been  tested.) 

The  TADS/PNVS  transition  to  production  phase  was  successful  because  the 
system  was  at  a  sufficiently  mature  technology  readiness  level  for  the  transition.  A  lot  of 
work  was  needed  to  meet  remaining  performance  requirements  and  on  manufacturability 
and  reliability,  but  most  requirements  already  had  been  met  at  this  point.  The  developer 
and  many  Government  labs  contributed  to  the  success  of  the  TADS  /  PNVS  system. 
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E.  RECOMMENDATIONS  FOR  FURTHER  STUDY 


1.  Technology  Maturation: 

Study  the  various  processes  of  technology  maturation,  including  technology 
developed  for  civilian  industry,  Research  Development  partnering  development  efforts  by 
Defense  contractors,  and  technology  developed  specifically  for  a  program.  Study  the 
technical  and  financial  processes,  and  the  risk  to  overall  project  funding. 

2.  Technology  Readiness: 

Study  technology  readiness  at  the  beginning,  during,  and  after  each  of  the  major 
wars  of  this  century  (World  War  I,  World  War  II,  Korean  War,  and  the  Viet  Nam  War) 
and  compare  these  readiness  levels  to  those  of  Operation  Desert  Storm  and  the  war  in 
Afghanistan. 


3.  Technology  Readiness  Level  Measurement: 

Study  various  methods  used  to  measure  technology  readiness  level,  by  whatever 
names  they  are  known,  and  how  well  each  technique  works  to  give  the  program  manager 
knowledge  of  the  status  of  the  program. 
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III.  DEVELOPMENT  ORGANIZATIONS 


A.  RESEARCH  QUESTION 


This  chapter  answers  the  secondary  research  question,  “How  were  the 
organizations  that  had  developed  these  critical  technologies  involved  during  system 
development?”  This  chapter  will  look  at  the  involvement  of  the  organizations  that  had 
developed  the  critical  technologies  during  system  development,  in  terms  of  (1)  the  role  of 
S&T  organization  that  developed  technology,  (2)  role  of  Government  S&T  organization, 
(3)  difficulties  in  integrating  technology,  (4)  production  readiness,  (5)  importance  of 
technology  to  Prime  Contractor,  (6)  familiarity  of  Prime  with  technology,  and  (7)  timely 
problem  disclosure.  I  will  list  the  data,  then  analyze  it,  and  finally  draw  conclusions. 


17.  RADAR  FREQUENCY  INTERFEROMETER  0TFI)  H  26. 

18.  AFT  STORAGE  BAY  27. 

19.  MAI  II  LANDING  GEAR  28. 

2D.  UTILITY  LIGHT  AIIDGROUND  POWER  OUTLET  ACCESS  DOOR  29 

21.  TADS/PNVS  TURRET  30. 

22.  RIGHT  SIDE  EFAB  31. 

23.  FIRE  EXTINGUISHER  ACCESS  DOOR  32. 

24.  INTERCOMMUNICATIONS  ACCESS  DOOR  33. 

25.  TRANSMISSION  ACCESS  DOOR  34. 

35. 


ENGINE  NACELLE  ASSEMBLY 

ANTI  COLLISION  LIGHTS  AND  NAVIGATION  LIGHTS 

HYDRAULIC  GROUND  SERVICE  PANEL  ACCESS  DOOR 

HORIZONTAL  STABILIZER 

VERTICAL  STABILATOR 

GLOBAL  POSITIONING  SYSTEM  (GPS)  ANTENNA 

ICE  DETECT  PROBE 

FLAT  PLATE  (FCR  REMOVED! 

RIGHT  FLYAWAY  KIT  BAY 
LEFT  FLYAWAY  KIT  BAY 


LBA-0002 


Figure  2-1.  General  Arrangement  (Sheet  2  of  2) 


Figure  3.  General  Arrangement  page  2,  fromTM  1-1520-251-10,  Operator’s  Manual  for 
Helicopter,  Attack,  AH-64D,  Longbow  Apache,  15  Dec  1998 
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B. 


DATA 


1.  Role  of  S&T  Organization  that  Developed  Technology? 

This  covers  survey  questions  T8,  T9,  T10,  and  Page  1.  Page  1  data  is  listed  in 


Chapter  II. 


T8.  For  each  of  the  technologies  A,  B  &  C, 
did  an  Army  Laboratory  or  Center  make  a 
significant  contribution  to  achieving  any  of 
the  above  levels  of  technology  readiness? 

Technology 

A 

Technology 

B 

Technology 

C 

T8.  Yes,  it  contributed  to  Readiness  at 
start  of  Planning/Pre-development. 

X 

T9.  Yes,  it  contributed  to  Readiness  for 
Development. 

X 

X 

X 

T10.  Yes,  it  contributed  to  Readiness  for 
Transition  to  Production. 

X 

X 

X 

Table  8.  Data  for  Questions  T8,  T9,  and  T10 


2.  Role  of  Government  S&T  Organization? 

This  covers  survey  questions  T8  through  T10,  B 1 1 ,  and  Page  1.  Page  1  data  is 
listed  in  Chapter  II;  and  T8  through  T10  are  listed  above  in  ‘2.  Role  of  S&T  organization 
that  developed  technology?’ 

B 1 1 .  Army  Labs/Centers  resisted  project  ideas  or  approaches.  X  (No) 

3.  Difficulties  in  Integrating  Technology? 

This  covers  survey  questions  T3,  H3,  Bl,  and  B4  through  B8. 

T3.  Production  Impact:  What  was  the  impact  of  the  technology  on  then 
existing  production  processes? 


(Answer  for  date  you  provided  for  Development  start,  I 

>.) 

Technology 

A 

B 

c 

1.  Technology  forced  deep  and  serious  production  process  change? 

X 

X 

2.  Technology  caused  significant  production  process  change? 

X 

Table  9.  Data  for  Question  T3 


This  level  was  chosen  because  the  contractor  was  not  producing  other  systems 
like  this  at  the  time.  At  the  component  level,  production  processes  were  not  significantly 
different  and  did  not  require  much  change;  however,  and  the  system  and  major 
subsystem-level  (FLIR,  Day  Sight,  bore-sight)  production  acceptance  test  stations  had  to 
be  created  to  insure  that  delivered  hardware  was  meeting  system-level  specifications. 
Also,  an  effort  was  made  to  identify  component  tests  and  processes,  which  would  reduce 
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the  failures  that  would  be  seen  at  system-level  and  major  subsystem-level  acceptance 
tests.  These  component  tests  were  unique  and  different  some  of  the  times  because  they 
were  driven  bv  svstem-level  specifications,  which  were  unique  at  this  time  to  the 
TADS/PNVS. 

H3.  Key  Skills.  This  question  asks  about  “key  skills”  essential  to  the  success  of 
the  project,  defined  as  skills  “that  if  they  were  not  available  at  all,  would  have  stopped 
team  progress  at  the  point  when  they  were  needed.” 

Were  there  any  key  skills  not  adequately  represented  on  the  team?  X  No. 

The  design  chief  could  draft  people  from  other  groups.  It  was  as  good  as  "DX 
brick  bat”  priority,  which  same  individual  had  later  on,  at  least  within  the  company  in 
Orlando.  However,  they  didn ’t  have  the  microwave  electronics  hybrids  design  group, 
nor  the  printed  circuit  layout  design  people  on  the  project.  Those  were  both  functional 
groups,  and  the  TADS/PNVS  group  didn ’t  have  enough  work  to  justify  keeping  them  on 
their  team.  But  they  had  as  good  of  a  priority  with  these  groups  as  any  other  project  in 
the  company  in  Orlando. 

Bl.  It  was  harder  than  expected  to  take  the  risk  out  of  the  new  technology. 
Major  effort 

B4.  A  critical  production  issue  was  uncovered  very  late  in  the  process.  Minor 

effort 

B5.  Management  pressure  pushed  technology  prematurely  into  production. 
Minor  effort 

B6.  There  was  a  lack  of  acceptance  standards  for  the  new  technology.  Very  minor 

effort 

B7.  The  technology  was  hard  to  scale  up  from  lab  &  pilot  tests.  Significant  effort 
B8.  Testing,  quality  control  and/or  acceptance  took  longer  than  planned. 
Significant  effort 

4.  Production  Readiness? 

This  covers  survey  questions  Page  1,  T3,  H6,  B4,  B5,  B6,  and  B8.  Page  1  data  is 
listed  in  Chapter  II;  T3,  B4,  B5,  B6,  and  B8  are  listed  above  in  ‘4.  Difficulties  in 
integrating  technology?’ 

H6.  Whose  facilities  were  going  to  be  the  primary  production  site  for  the 

application  of  the  new  technologies?  X  1.  Prime  contractor’s  facilities  _ 2.  Both 

Prime  and  supplier  facilities  _ 3.  Supplier  facilities 

5.  Importance  of  Technology  to  Prime? 

This  covers  survey  questions  Page  1  and  T4.  Page  1  data  is  listed  in  Chapter  II. 
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Check  (si)  the  best  answer  for  each 
technology. 

Technologv  A 

Technologv 

B 

Technologv 

C 

T4.  Looking  back  at  the  Development  start  date,  at  that  time  how  important  were  these 
technologies  to  the  Prime? 

Prime  was  planning  or  had  started 
follow-on  uses  of  the  technology. 

2.  X 

2.  X 

2.  X 

Table  10.  Data  for  Question  T4 


6.  Familiarity  of  Prime  with  Technology? 

This  covers  survey  questions  Page  1,  T2,  and  T3.  Page  1  data  is  listed  in  Chapter 

II;  and  T3  is  listed  above  in  ‘4.  Difficulties  in  integrating  technology?’ 

T2.  How  new  was  each  technology  to  the  Prime  Contractor?  For  each 
technology  A,  B,  and  C,  was  the  technology: _ _ _ _ 


Technology 

A 

B 

c 

1 .  New  and  unproven  for  the  Prime  Contractor? 

X 

X 

2.  Technology  had  been  used  by  Prime  Contractor,  but  it  was 

new  to  this  kind  of  application? 

X 

Table  1 1 .  Data  for  Question  T2 


7.  Timely  Problem  Disclosure? 

This  covers  survey  questions  D12,  D16,  and  D19. 

D12.  The  team  was  reluctant  to  share  concerns  with  Government  PM.  1  X 
(Strongly  disagree) 

D16.  Usually  team  knew  right  away  where  to  get  necessary  outside  help.  4  X 
(Agree  somewhat) 

D19.  The  Government  PM  was  reluctant  to  share  problems  with  Anny  leaders.  1 
X  (Strongly  disagree) 

C.  ANALYSIS 

This  section  will  analyze  the  involvement  of  the  organizations  that  had  developed 
the  critical  technologies  during  system  development  in  terms  of  (1)  role  of  S&T 
organization  that  developed  technology,  (2)  role  of  Government  S&T  organization,  (3) 
difficulties  in  integrating  technology,  (4)  production  readiness,  (5)  importance  of 
technology  to  Prime,  (6)  familiarity  of  Prime  with  technology,  and  (7)  timely  problem 
disclosure. 
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1.  Role  of  S&T  Organization  that  Developed  Technology? 

The  U.S.  Army  Night  Vision  Labs  (NVL)  was  the  original  developer  of  the  FLIR 
technology  used  in  the  TADS  night  sight  and  the  PNVS.  They  provided  support  to  the 
TADS/PNVS  program  from  the  very  start  of  the  System  Planning  phase,  and  they 
continued  to  provide  support  through  development  and  the  Transition  to  Production 
phase. 

The  MICOM  Guidance  and  Control  (G&C)  Lab  was  involved  in  system 
requirements  for  Total  Pointing  Error  (TPE)  for  the  laser  designator,  which  is  a 
component  of  laser  to  sensor  bore-sight.  G&C  labs  did  a  lot  of  testing  and  simulation 
work  to  develop  these  requirements  and  early  work  on  the  laser  hardware. 

Martin  Marietta  Corporation  Science  and  Technology  organizations  were  doing 
their  own  work  in  response  to  the  anticipated  requirement  for  the  ASH  and  AAH 
programs.  They  needed  to  develop  the  technology  and  to  create  a  manufacturing  plan. 

2.  Role  of  Government  S&T  Organization? 

In  addition  to  the  involvement  listed  above,  Frankfort  Arsenal,  as  well  as  U.S. 
Army  Night  Vision  Labs  (NVL)  and  MICOM  Guidance  and  Control  (G&C)  Lab,  gave 
significant  support  in  fire  control  and  optics  in  developing  requirements,  evaluating 
proposals  and  monitoring  development  progress.  These  labs  were  quite  open  to 
requirements  changes  and  other  project  ideas. 

Army  labs  contributed  to  readiness  at  the  start  of  the  planning  phase  for  LLIR 
target  acquisition.  They  continued  to  provide  readiness  support  for  the  three  critical 
technologies  throughout  development  and  the  transition  to  production  phases. 

3.  Difficulties  in  Integrating  Technology? 

The  contractor  had  to  make  serious  changes  in  their  production  process  for  two  of 
the  three  most  critical  technologies  (LOSS  and  FLIR  Target  Acquisition)  and  significant 
changes  for  the  third  (Laser  to  sensor  bore-sight).  The  contractor,  Martin-Marietta,  was 
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not  then  producing  similar  systems.  Components  were  similar,  but  new  types  of  system 
tests  had  to  be  developed  in  order  to  guarantee  meeting  system  specifications. 

Using  a  novel  testing  philosophy  to  find  system  faults  earlier,  some  requirements 
were  flowed  down  to  lower  level  modules  and  components  to  eliminate  failures  earlier  in 
the  process.  These  tests  were  often  unique  because  they  were  driven  by  system-level 
requirements. 

The  TADS  /  PNVS  was  a  critical  contract  for  Martin-Marietta,  and  the  upper 
management  put  a  high  priority  on  it.  They  provided  personnel  in  adequate  numbers  with 
the  skills  needed  for  the  project.  Some  specialties  were  from  functional  groups  that  gave 
the  TADS/PNVS  group  a  high  priority,  but  didn’t  transfer  personnel  -  because  their  full 
time  services  were  not  necessary. 

Various  risk  factors  caused  the  program  major  difficulties.  For  example,  taking 
the  risk  out  of  the  new  technologies  was  a  major  effort.  Also,  significant  effort  was 
needed  both  to  scale  the  technology  up  from  lab  and  pilot  tests  and  to  run  tests 
successfully.  However,  only  minor  effort  was  needed  to  deal  with  critical  production 
issues,  with  management  pressure  pushing  technology  too  quickly  into  production,  and 
with  the  lack  of  acceptance  standards  for  the  new  technologies. 

4.  Production  Readiness? 

Because  the  Prime  Contractor’s  facility  was  the  planned  production  site,  there  was 
no  need  to  transfer  the  technology  to  a  new  facility,  with  the  consequent  learning  curve. 
A  sizable  portion  of  the  development  was  done  by  the  Prime  Contractor,  so  they  already 
had  a  lot  of  experience  with  these  technologies. 

The  TADS/PNVS  was  ready  for  production.  Some  of  the  risk  factors  (listed  in 
paragraph  4  above)  such  as  scaling  technology  and  running  tests  successfully,  slowed  the 
program  down  and  took  considerable  effort  to  overcome.  However,  other  factors 
required  only  minor  effort. 

The  three  critical  technologies  forced  significant  or  even  serious  production 
process  changes,  however  these  changes  were  not  all  unexpected  since  the  developer  was 
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also  the  production  company.  Some  of  these  changes  did  cause  some  delay  (about  6 
months)  in  production.  Components  were  similar  to  other  production  systems,  but  the 
system  was  not.  The  system-level  tests  forced  them  to  try  to  reduce  failures  by  instituting 
unusual  component  tests  to  catch  system  failures  earlier. 

5.  Importance  of  Technology  to  Prime? 

At  the  time  of  the  start  of  Development,  the  Prime  was  planning  or  had  actually 
started  follow-on  uses  of  all  three  critical  technologies:  Line-of-Sight  Stabilization 
(LOSS),  FLIR  target  acquisition  (FTA),  and  Laser  to  sensor  bore-sight  (LSBS).  Martin 
Marietta  did  have  some  follow-on  contracts  that  made  use  of  this  technology  (e.g.  U.S. 
Air  Force  LANTIRN).  Many  problems  had  to  be  overcome  to  get  the  TADS  /  PNVS 
operational;  but  the  knowledge  gained  helped  the  Prime  establish  itself  in  this  technology 
and  gain  a  foothold  in  a  profitable  market. 

6.  Familiarity  of  Prime  with  Technology? 

The  Laser  to  sensor  bore-sight  and  Line-of-Sight  (LOS)  Stabilization  were  new 
and  unproven  technologies  for  the  Prime  Contractor,  Martin  Marietta.  They  had  used 
FLIR  target  acquisition,  but  they  were  new  to  this  kind  of  application.  The  contractor 
struggled  quite  a  bit  in  getting  this  technology  working. 

Technology  forced  deep  and  serious  production  process  changes  for  both 
stabilization  and  bore  sighting.  FLIR  target  acquisition  required  significant  production 
process  changes.  Production  acceptance  test  stations  for  these  technologies  were  created 
to  test  hardware  to  the  system-level  specifications.  They  tried  to  identify  component  tests 
and  processes  that  would  catch  both  system-level  failures  and  major  subsystem  failures. 
The  component-level  tests  were  unique  in  that  they  were  developed  to  find  system-level 
failures. 
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7. 


Timely  Problem  Disclosure? 


When  there  were  problems,  usually  the  development  team  knew  immediately 
where  to  get  outside  help.  The  development  team  was  open  about  sharing  concerns  with 
the  Government  PM,  and  the  PM  shared  problems  with  Anny  leaders.  This  open 
communications  helped  the  Government  stay  informed  and  fix  problems  before  they 
became  too  big.  Any  problems  the  team  couldn’t  handle  directly,  or  with  help  they  could 
get,  the  Army  was  in  a  position  to  know  about  the  problem  and  take  steps  to  resolve  it. 


D.  CONCLUSIONS 

This  section  draws  conclusions  concerning  the  involvement  of  the  organizations 
that  had  developed  these  critical  technologies  during  system  development. 

1.  Role  of  S&T  Organization  that  Developed  Technology? 

The  U.S.  Army  Night  Vision  Labs  (NVL),  the  MICOM  Guidance  and  Control 
(G&C)  Lab,  and  Martin  Marietta  Corporation  Science  and  Technology  organizations,  all 
of  which  developed  important  TADS/PNVS  technology,  were  actively  supporting  the 
program  with  reviews  and  additional  lab  work. 

2.  Role  of  Government  S&T  Organization? 

Additionally,  Frankfort  Arsenal  gave  further  technical  support.  These  labs  gave 
assistance  in  the  areas  of  requirements  changes  and  they  were  open  to  other  project  ideas 
throughout  the  development  and  transition  to  production  phases. 


3.  Difficulties  in  Integrating  Technology? 

Martin  Marietta  made  significant  changes  to  accommodate  production  of 
TADS/PNVS,  in  their  process  and  in  new,  more  stringent  component  tests.  TADS/PNVS 
was  able  to  get  most  of  the  personnel  they  needed  pennanently  on  their  team,  and  high 
priority  for  some  functional  specialties  that  were  needed  for  only  part  of  the  time. 
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Major  effort  was  needed  to  take  the  risk  out  of  the  new  technologies,  although  the 
project  team  was  eventually  successful;  and  significant  effort  was  needed  both  to  scale 
the  technology  up  from  lab  and  pilot  tests  and  to  run  tests  successfully.  However,  only 
minor  effort  was  needed  to  deal  with  other  development  and  production  problems. 

4.  Production  Readiness? 

The  Prime  Contractor  was  ready  for  production,  since  they  also  participated  in 
development.  Some  technological  risk  factors  took  some  time  and  effort  to  overcome, 
but  nothing  out  of  the  ordinary.  There  were  production  process  changes  required  by  the 
three  critical  technologies,  but  delays  were  minimal  -  about  six  months.  System-level 
testing  also  caused  some  additional  production  readiness  problems. 

5.  Importance  of  Technology  to  Prime? 

The  critical  technologies  in  the  system  were  of  great  value  to  the  developer,  both 
for  TADS/PNVS  contracts,  and  for  other  follow-on  contracts.  The  problems  that  the 
Prime  overcame  established  it  in  a  profitable  market. 

6.  Familiarity  of  Prime  with  Technology? 

The  critical  technologies  of  the  TADS/PNVS  system  were  mostly  new  to  the 
developer  at  the  start  of  the  program,  causing  some  struggle  to  master  these  technologies. 
FLIR  target  acquisition  had  been  used  before,  but  in  a  dissimilar  application.  This  high 
technology  also  forced  production  changes  to  their  factory.  In  addition,  Martin-Marietta 
adopted  new  subsystem  and  component  testing  to  ferret  out  system-level  problems. 

7.  Timely  Problem  Disclosure? 

Problems  were  freely  reported  from  developer  to  Government  PM,  and  from  PM 
to  Army  leaders.  This  open  communications  helped  the  Government  stay  informed  and 
fix  problems  before  they  became  too  big.  If  a  problem  occurred  which  was  outside  team 
members’  capabilities,  the  team  was  always  able  to  get  outside  help. 
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Development  Organizations:  In  summary,  both  Government  agencies  and  the 
developer  contributed  greatly  to  the  success  of  the  TADS  /  PNVS  program.  Significant 
effort  was  needed  to  develop  the  system,  in  some  cases  major  effort.  Significant  effort 
also  was  needed  for  production  readiness.  But  the  new  technology  field  of  TADS/PNVS 
was  a  strong  motivator  to  Martin  Marietta.  The  critical  technologies  were  mostly  new  to 
the  developer,  but  their  effort  paid  off.  And  any  problems  they  encountered  were  freely 
reported  by  the  developer  to  the  PMO,  and  by  the  PMO  to  Army  higher  headquarters, 
which  allowed  additional  resources  to  be  used  to  head  off  some  potential  problems. 


E.  RECOMMENDATIONS  FOR  FURTHER  STUDY 

1.  Market  Share  Over  Time: 

Examine  the  market  share  that  Martin-Marietta  Corporation,  now  Lockheed 
Martin  Corporation,  enjoyed  with  the  three  critical  technologies  over  time. 

2.  Technology  Buy-In: 

Examine  how  some  companies  buy  into  certain  technologies,  by  buying  a 
company  in  the  field  or  by  bidding  below  cost  on  a  contract,  and  whether  the  venture  was 
financially  successful  for  the  company  in  the  long  run.  Also,  examine  the  effect  on  their 
customer(s)  of  using  ’novices'  in  this  technical  area. 

3.  Science  and  Technology  Role: 

Examine  scientific  groups  in  various  companies,  and  how  they  contribute  to 
developing  financially  successful  products. 
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IV.  USER  SUPPORT  AND  FUNDING  STABILITY 


A.  RESEARCH  QUESTION 

This  chapter  answers  the  research  question,  “To  what  extent  was  there  user 
support  and  funding  stability  during  system  development?”  This  chapter  will  cover  the 
extent  of  user  support  and  funding  stability  during  system  development  in  terms  of  (1) 
user  support  (or  role  of  user),  (2)  requirements  stability,  and  (3)  funding  stability.  I  will 
introduce  the  data,  then  analyze  it,  and  finally  draw  conclusions. 


Figure  4.  TADS  Sighting  System  Image,  from  TM  1-1520-251-10,  Operator’s  Manual  for 
Helicopter,  Attack,  AH-64D  Longbow  Apache,  15  Dec  1998 

B.  DATA 

1.  User  Support?  (Or  Role  of  User?) 

This  covers  survey  questions  D18,  F5,  F6,  W3,  W4,  and  W5. 

D18.  There  was  a  lot  of  contact  with  TRADOC*  during  the  project.  Strongly  Agree 
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*By  TRADOC  here  and  elsewhere,  we  mean  Training  &  Doctrine  Command  and/or  other 
appropriate  user  representatives. 


How  often  did  the  following  occur  during  i 

Development? 

Questions  F5-F7  use  the  following 
possible  answers: 

Never 

Once 

or 

Twice 

Several 

times 

Many 

Times 

Don’t 

know 

N/A 

F5.  Did  TRADOC/other  user  organizations  show  strong  support?  Many  Times 


F6.  Were  there  changes  in  key  TRADOC  or  other  user  personnel?  Once  or  Twice 


Please  check  (*Q  all  stages  when  the  activity  occurred. 


SP 

D 

TP 

Selection 

Development 

Transition 

(DK/ 

N/A) 

Early 

Middle 

Later 

(Never) 

W3.  When  was  the  TRADOC 
consulted  on  project  questions? 

X 

X 

X 

X 

X 

W4.  When  was  there  change  in 
key  TRADOC  /  user 
representatives? 

X 

W5.  When  did  TRADOC  /  other 
users  show  strong  support? 

X 

X 

X 

X 

X 

W6.  When  was  there  change  in 
the  system  requirements? 

X 

Table  12.  Data  for  Questions  W3,  W4,  W5,  and  W7 


“The  TRADOC  Systems  Manager  and  other  military  personnel  changed  about 
every  three  years...  However,  I  don ’t  think  this  was  ever  a  problem. 


2.  Requirements  Stability? 

This  covers  survey  questions  F7,  W6,  and  B13.  W6  is  listed  above  with  W3,  W4, 
and  W5  for  legibility. 

How  often  did  the  following  occur  during  Development? 

F7.  Were  there  changes  in  system  requirements  (e.g.,  threat)?  Never 

Did  this  problem  come  up  during  this  project? 

B13.  Threat  definition  or  other  requirements  changed  during  the  project.  X  No 
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3.  Funding  Stability? 

This  covers  survey  questions  HI,  D1 1,  and  B2. 

HI.  At  some  point,  was  the  project  either  (slowed  down  or  stopped  and 
restarted)?  X  3.  Neither 

The  TADS/PNVS  program  was  originally  part  of  the  ASH  (Advanced  Scout 
Helicopter)  program,  which  was  cancelled.  This  happened  prior  to  1977,  the  start  of  SP 
phase.  AAH  (Apache  AH  64  PMO)  was  already  involved  when  ASH  left  the  program,  as 
was  MICOM.  AAH  and  MICOM  support  of  TADS/PNVS  continued  on,  after  ASH  left  the 
program. 

Dll.  There  was  often  uncertainty  about  the  future  of  project  funding.  Strongly 

agree 

B2.  Cutbacks  in  project  resources  forced  changes/compromises.  Very  minor 

effort 

C.  ANALYSIS 

This  section  analyzes  the  data,  comparing  the  two  individual  responses,  as  well  as 
advantages  and  disadvantages,  analyzing  them  in  terms  of  the  primary  and  secondary 
questions.  This  section  will  analyze  the  extent  there  was  user  support  and  funding 
stability  during  system  development  in  terms  of  (1)  User  support?  (Or  role  of  user?),  (2) 
Requirements  stability,  and  (3)  Funding  stability. 


1,  User  Support?  (Or  Role  of  User?) 

The  TADS/PNVS  Program  Office  had  a  lot  of  contact  with  the  Training  & 
Doctrine  Command  (TRADOC)  during  development.  TRADOC  is  the  primary  interface 
to  the  users.  TRADOC  frequently  showed  strong  support  of  the  project.  Occasionally, 
there  were  changes  in  key  TRADOC  personnel,  approximately  every  three  years,  but  this 
never  affected  the  program  much. 

TRADOC  was  consulted  on  project  questions  throughout  the  program,  from 
earliest  systems  planning,  through  development,  and  into  the  transition  to  production. 
And  TRADOC  responded  by  showing  strong  support  for  the  TADS/PNVS  program 
throughout  the  same  period. 
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2.  Requirements  Stability? 

The  system-level  requirements  were  very  stable  during  development.  The  threat 
definitions  (detail  requirements)  that  the  TADS/PNVS  was  required  to  counter  were 
stable,  as  well.  Requirements  changes  in  the  development  period  can  radically  change 
the  design.  Sometimes  the  contractor  has  to  get  extra  money  or  time  to  effect  these 
changes. 


3.  Funding  Stability? 

Project  funding  was  frequently  uncertain.  The  project  required  almost  twice  the 
contracted  amount,  and  the  extra  money  had  to  be  provided  by  the  AAH  Program 
Manager. 

The  project  usually  had  all  the  resources  needed  for  development.  Occasionally, 
some  minor  effort  was  needed  to  make  changes  or  compromises  because  of  resource 
shortages. 

Although  the  Advanced  Scout  Helicopter  (ASH)  program,  which  was  leading  the 
TADS  /PNVS  program,  was  cancelled,  the  AAH  (Apache  AH-64  PMO)  was  already 
involved  as  was  MICOM.  There  was  really  no  affect  on  the  program,  other  than  a  change 
in  leadership.  Also,  instead  of  needing  to  meet  the  demands  of  two  PMOs,  the  developer 
now  only  had  to  satisfy  one,  which  lowered  the  technical  risk. 


D.  CONCLUSIONS  ON  USER  SUPPORT  AND  FUNDING  STABILITY 

This  section  will  draw  conclusions  concerning  the  extent  there  was  user  support 
and  funding  stability  during  system  development. 

1.  User  Support?  (Or  Role  of  User?) 

The  Training  &  Doctrine  Command  (TRADOC)  provided  strong  support  to  the 
TADS/PNVS  Program  Office  during  development.  TRADOC  was  consulted  on  the 
program,  and  provided  worthy  user  representation. 
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2.  Requirements  Stability? 

TADS/PNVS  program  had  good  requirements  stability  at  the  system  level  and 
stable  threat  definitions. 


3.  Funding  Stability? 

Funding  was  quite  unstable  on  the  TADS/PNVS  program.  However,  they  usually 
had  most  of  the  resources  they  needed  when  funding  was  stable.  Although  the  Advanced 
Scout  Helicopter  (ASH)  program  was  cancelled,  this  affected  neither  program  funding, 
nor  program  continuity. 


E.  RECOMMENDATIONS  FOR  FURTHER  STUDY 

1.  Funding  Stability  and  Program  Effectiveness: 

Study  various  programs  with  good  and  poor  funding  histories,  and  how  the 
program  has  been  effective  in  developing  a  useful  product. 

2.  Requirements  Stability  and  Program  Effectiveness: 

Study  various  programs  with  good  and  poor  requirements  stability,  and  how  the 
program  has  been  effective  in  developing  a  useful  product. 

3.  User  Support  and  Program  Effectiveness: 

Study  various  programs  with  good  and  poor  user  support,  and  how  the  program 
has  been  effective  in  developing  a  useful  product. 
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V.  INTEGRATED  PRODUCT  TEAMS 


A.  RESEARCH  QUESTION 

This  chapter  answers  the  research  question,  “How  effectively  were  (what  we  now 
call)  integrated  product  teams  (IPT)  employed  during  development?”  This  chapter  will 
look  at  the  effectiveness  of  IPTs  in  terms  of  (1)  IPT  approach  used,  (2)  Proper  staffing  of 
IPT,  (3)  Design  to  manufacturing  linkage,  and  (4)  Design  to  supplier  linkage.  I  will 
introduce  the  data,  then  analyze  it,  and  finally  draw  conclusions. 


OPTICAL  RELAY  TUBE 


TADS/PNVS  DISPLAY 

TURRETS  ELECTRONICS  TADS 

UNIT  POWER 

SUPPLY 


USER 

1  ELECTRONICS  UNIT 


LBA- 


Fiqure  4-1.  Siqht  Subsystem 

Figure  5.  TADS  /  PNVS  Sight  Subsystem,  from  TM  1-1520-251-10,  Operator’s  Manual  for 
Helicopter,  Attack,  AH-64D,  Longbow  Apache,  15  Dec  1998 
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B.  DATA 

1.  IPT  Approach  Used? 

This  covers  survey  questions  H2,  H4,  H5,  Dl,  D7,  D9,  D13,  D14,  D16,  D19,  and 
F4. 

H2.  Was  the  project  set  up  as  a  cross-functional  integrated  product  team  (IPT),  a 
project  team  drawn  from  different  parts  of  the  contractor’s  organization  with  most  of  the 
skills  needed  for  the  development?  Yes. 

If  YES,  was  it:  X  1.  Set  up  by  management,  with  different  functions  & 
departments  tasked  to  provide  team  members. 

In  the  interview,  the  government  respondent  goes  on  to  explain  that  they  did  not 
call  them  IPTs  then,  but  they  were  essentially  the  same  thing. 

“77?/. v  concept  of  an  integrated  product  team  (IPT),  really  came  more  into  vogue 

about  or  after  the  time  we  first  went  into  production.  At  that  time  there  was  a  big 
emphasis  to  bring  in  production  people,  logistics,  and  so  forth  in  the  very  early  stages  of 
the  design  program.  In  the  early  stages  of  the  TADS  PNVS  program,  we  did  have  those 
people  involved. 

We  did  not  call  it  IPT  and  they  didn  7  organize  it  that  much,  but  there  were 
reliability,  logistics,  maintenance,  and  production  requirements.  In  the  beginning  of  the 
program  in  1977,  when  they  did  this  initial  primary  design  with  seven  different 
contractors  and  then  the  fly-off  there  was  much  heavier  emphasis  ...  on  the  performance 
aspects  of  TADS/PNVS,  because  it  is  something  that  no  one  had  ever  done  before,  so  the 
rest  of  (the  program)  doesn't  matter  if  you  cannot  do  the  performance  part." 


H4.  During  the  Development  stage  of  the  project,  how  many  people  on  the 
team  were  collocated  very  close  together?  (On  the  same  floor  of  a  building  within  a  one- 
minute  walk.)  X  2.  Most  (2/3rds  or  more) 

H4a.  Including  the  above,  how  many  people  on  the  team  were  collocated 
in  the  same  building?  X  2.  Most  (2/3rds  or  more) 

Most  were  in  the  same  building,  about  90%,  and  the  rest  were  in  another  building 
in  the  same  city. 

H5.  How  many  people  on  the  team  involved  in  the  Development  stage  had 
worked  before  with  others  on  the  project?  X  2.  Most  (2/3rds  or  more) 

Team  Participants  &  Communications  during  Development  (D1-D19) 

Here  are  some  statements  about  the  people  on  the  project  during  the  System 
Development  stage.  Please  circle  a  number  to  indicate  your  level  of  agreement  or 
disagreement  that  each  statement  is  a  description  of  team  processes  on  this  project. 
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Dl.  The  team  leader  was  good  at  resolving  technical  disagreements.  Strongly 

Agree 

D7.  Team  meetings  were  sometimes  frustrating  and  non-productive.  Neither 
agree  nor  disagree 

D9.  Project  results  did  not  take  advantage  of  the  team’s  best  ideas.  Disagree 
somewhat 

D13.  Management  project  reviews  were  constructive  &  helpful.  Agree  somewhat 
D14.  Formal  reviews  were  conducted  at  key  decision  points.  Strongly  Agree 
D16.  Usually  team  knew  right  away  where  to  get  necessary  outside  help.  Agree 
somewhat 

D19.  The  Government  PM  was  reluctant  to  share  problems  with  Army  leaders. 
Strongly  Disagree 

How  often  did  team  members  do  the  following  during  Development? 

F4.  Needed  management  help  to  resolve  project  team  disagreements?  X  (Once 
or  Twice) 

2.  Proper  Staffing  of  IPT? 

This  covers  survey  questions  H3,  D3  through  D6,  D8,  and  DIO.  H3  data  is  in 
Chapter  III. 

D3.  There  was  a  lot  of  turnover  in  team  membership.  Disagree  somewhat 

D4.  The  team  leader  had  both  design  &  production  experience.  Neither  agree  nor 
disagree.  Developer  team  leader  had  excellent  design  experience;  but  production 
experience  was  associated  with  smaller  systems. 

D5.  The  team  leader  had  very  high  technical  competence.  Strongly  agree 

D6.  Some  key  technical  skills  were  not  represented  on  the  team  itself.  Disagree 
somewhat 

D8.  Professionals  were  split  across  too  many  different  tasks  &  teams.  Neither 
agree  nor  disagree 

DIO.  Key  members  continued  through  pre-production  planning  and  testing.  Agree 
somewhat 


3.  Design  to  Manufacturing  Linkage? 

This  covers  survey  questions  FI,  F2,  F3,  F10  through  F13,  Wl,  W2,  W16,  W17, 
and  W18. 


Questions  F 1  throug] 

h  F13  use  the  following  responses: 

Never 

Once  or 
Twice 

Several 

times 

Many 

Times 

Don’t  know 

Not  Applicable 
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FI.  Went  to  the  shop  floor  to  meet  about  related  production  processes.  Many 

Times 

F2.  Asked  for  supplier  comments  &  suggestions  on  design  choices.  Several  times 
F3.  Showed  &  discussed  physical  models  of  new  components  with  suppliers. 
Once  or  Twice 

F10.  Passed  around  physical  prototypes  during  joint  discussions.  Many  Times 
Fll.  Held  planning  meetings  that  included  both  design  &  production  people. 
Once  or  Twice 

F12.  Explored  choices  together  with  computational  models  or  analytic  tools. 
Never  The  Manufacturing  engineers  reviewed  prints  all  throughout  the  project,  hut 
they  didn ’t  use  computational  models  or  analytic  tools.  Computer  tools  didn ’t  exist  at  the 
time,  and  they  didn ’t  have  any  manufacturing  analytical  tools  -  1970s  and  early  1980s. 

F13.  Had  test  articles  or  pre-production  parts  to  discuss  and  examine  jointly. 
Once  or  Twice 


Please  check  (W)  all  stages  when  the  activity  occurred. 


SP 

D 

TP 

Selection 

Development 

Transition 

(DK/ 

N/A) 

Early 

Middle 

Later 

(Never) 

W 1 .  When  did  production 
representatives  participate 
regularly? 

X 

X 

W2.  When  did  team  members 
meet  with  production  on  shop 
floor? 

X 

X 

Table  13.  Data  for  Questions  W1  and 

1 W2 

Relationship  &  Activities  between  Engineering  Design  &  Production/Program 
These  questions  are  different  because  they  focus  only  on  joint  meetings  or 
discussions  that  included  both  DESIGN  personnel  and  people  from  PRODUCTION 
and/or  PROGRAM  people  concerned  with  production. 
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Please  check  (*Q  all  stages  when  the  activity  occurred. 


SP 

D 

TP 

Selection 

Development 

Transition 

(DK/ 

N/A) 

Early 

Middle 

Later 

(Never) 

W16.  When  did  the  team  & 
technical  professionals  from 
Production  have  unscheduled  & 
informal  joint  conversations 
about  the  project? 

X 

X 

X 

W17.  When  were  analytic 
engineering  tools  used  jointly  by 
Design  and  Production  to 
explore  options  together? 

X 

W18.  When  were  prototypes  and 
parts  used  in  joint  discussions? 

X 

X 

Table  14.  Data  for  Questions  W16,  W17,  and  W18 


4.  Design  to  Supplier  Linkage? 

This  covers  survey  questions  F20  through  F23,  W26,  W27,  W28,  and  BIO. 


SHARED  DESIGN-SUPPLIER  ACTIVITIES  during  System  Development.  Now 
only  count  joint  meetings  or  discussions  that  included  personnel  from  both  DESIGN  and 
SUPPLIERS. 


Questions  F20  through  F23  use  the  following  responses: 


Never 

Once  or 

Several 

Many 

Don’t  know 

Twice 

times 

Times 

Not  Applicable 

F20.  Passed  around  physical  prototypes  during  joint  discussions.  Never 

F21.  Held  planning  meetings  that  included  both  design  and  suppliers.  Once  or 

Twice 

F22.  Explored  choices  together  with  computational  models  or  analytic  tools. 

Never 

F23.  Had  test  articles  or  pre-production  parts  to  discuss  and  examine  jointly. 
Many  Times 

Design  engineers  and  suppliers  worked  closely  together.  This  was  a  very  unique 
design  so  the  suppliers  were  designing/tailoring  their  hardware  for  this  specific  job  in 
many  cases. 
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Please  check  (*Q  all  stages  when  the  activity  occurred. 


SP 

D 

TP 

Selection 

Development 

Transition 

(DK/ 

N/A) 

Early 

Middle 

Later 

(Never) 

W26.  When  did  the  team  & 
technical  professionals  from 
Suppliers  have  unscheduled  & 
informal  joint  conversations 
about  the  project? 

X 

X 

X 

X 

W27.  When  were  analytic 
engineering  tools  used  jointly  bv 
Design  and  Suppliers  to  explore 
options  together? 

X 

W28.  When  were  prototypes  and 
parts  used  in  joint  discussions? 

X 

X 

Table  15.  Data  for  Questions  W26,  W27,  and  W28 


Did  this  problem  come  up  during  this  project? 

BIO.  One  or  more  suppliers  did  not  meet  their  commitments.  Significant  effort 
(was  needed) 

C.  ANALYSIS  OF  INTEGRATED  PRODUCT  TEAMS 

This  section  will  analyze  the  effectiveness  of  Integrated  Product  Teams  (IPT)  in 
tenns  of  (1)  IPT  approach  used,  (2)  Proper  staffing  of  IPT,  (3)  Design  to  manufacturing 
linkage,  and  (4)  Design  to  supplier  linkage. 


1.  IPT  Approach  Used? 

This  was  before  the  advent  of  the  formally-recognized  IPT  system  that  has  totally 
transformed  the  Government  and  business.  However,  there  was  then  a  realization  that 
integrating  people  from  many  disciplines  was  a  useful  technique.  Though  they  did  not 
call  them  IPT’s,  the  TADS  /  PNVS  program  frequently  used  multidisciplinary  working 
groups  to  solve  problems.  These  groups  were  not  formally  established,  though  people 
from  different  groups  were  invited.  This  often  happens  in  IPT’s  today  -  certain 
disciplines  may  not  be  represented  either  because  there  is  no  interest  or  due  to  lack  of 
funds  to  attend  IPT  meetings. 

A  similar  problem  both  then  and  today  is  that  often  the  membership  of  an  IPT 

varies.  The  membership  charter  may  call  for  one  (or  more)  person  from  a  specific 
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organization,  but  it  may  be  a  different  person  each  time.  If  this  happens,  there  is  no 
gradual  increase  in  either  the  working  relationship  between  members  or  the  skill  of 
members.  Different  people  from  the  same  organization  may  have  completely  different 
backgrounds  and  styles,  and  can  cause  disruption  when  they  contradict  previous  members 
of  their  own  organization.  These  changes  of  direction  can  be  very  disruptive. 

Also,  such  teaming  was  more  likely  on  critical  aspects  of  the  program. 
Performance  of  the  system  was  critical,  so  a  multidisciplinary  team  was  used. 

Multidisciplinary  work  is  not  confined  to  meetings  and  fonnal  groups.  Most  of 
the  people  on  the  program  were  in  the  same  building,  within  a  short  walk  of  each  other. 
This  fosters  quick,  informal  meetings  and  also  camaraderie  and  group  cohesion. 
Additionally,  many  people  had  worked  there  for  some  time,  even  before  the  project 
began.  Thus,  they  were  undoubtedly  experienced  with  working  together.  Some  people 
were  in  another  building  in  the  same  city,  so  it  was  not  too  difficult  to  have  face-to-face 
team  meetings  on  short  notice. 

The  success  of  any  team  depends  on  the  leadership  of  the  team  leader(s)  and  also 
the  skills  of  the  team  members.  During  development  of  the  TADS/PNVS,  the  team 
leader  was  good  at  resolving  technical  disagreements. 

But  the  path  can  be  rocky  in  arriving  at  agreement.  When  you  are  trying  to 
integrate  a  lot  of  technology  and  the  requirements  they  actualize,  there  are  often  trade¬ 
offs.  Compromising  can  be  difficult  for  some  people.  Occasionally,  someone  feels  that 
their  idea  must  take  precedence,  and  some  good  (competing)  ideas  can  be  lost.  Once  or 
twice,  it  was  necessary  to  get  management  help  to  resolve  disagreements. 

Usually  management  reviews  were  constructive.  They  had  formal  reviews  at  key 
decision  points.  The  Government  PM  reported  problems  that  went  up  to  Anny  leaders. 
Most  of  the  time,  it  was  easy  to  get  outside  help. 

In  the  days  before  IPTs,  there  were  lots  of  meetings.  These  meetings  may  not 
have  been  the  most  effective  solution  to  solving  problems,  but  they  did  solve  some.  The 
table  below  lists  a  range  of  types  of  Pre-IPT  Groups. 
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Type  of  Group 

Level  of  Analysis 

Staff  Meeting 

Infonnation  passing  /  Problem  solving 

Program  Status  Meeting 
(Dog  and  Pony  Show) 

Information  passing  -Very  high-level  / 
Critical  review 

Product  /  Functional  Status 
Meeting  (Low-level) 

Information  passing  -Lower-level  / 
Critical  review 

Working  Groups 

Problem  Solving  and  Information 
passing  -Lower-level 

Board  Approvals: 

Emergency  ECP 

Problem  solving  /  Critical  review 

Table  16.  1 

Vc-IPT  Groups 

These  groups  differ  from  each  other  in  the  level  of  the  group,  the  level  of  analysis, 
and  whether  they  include  both  Government  and  contractors.  Typical  staff  meetings  were 
simply  for  information  transfer,  mostly  downwards.  It  was  a  way  to  pass  the  word  to  the 
troops  with  the  least  work  for  the  chief.  But  occasionally,  there  were  problems  the  boss 
brought  up  and  people  would  work  on  them  together,  suggesting  strategies,  evaluating 
alternatives,  offering  related  information,  etc.  Both  Government  and  contractors  had 
their  own  staff  meetings,  typically  with  no  outsiders. 

The  typical  Program  Status  Meeting  (a.k.a.  “Dog  and  Pony  Show”)  was  a 
contractor  to  Government  interchange.  It  was  for  passing  very  high-level  information.  It 
really  was  not  possible  to  solve  many  problems  because  of  the  large  number  of  people 
present,  although  action  items  could  be  assigned. 

Product  /  Functional  Status  Meetings  were  more  low-level.  They  were  also  for 
information  passing,  but  at  a  lower  level.  Occasionally  they  were  conducted  like  working 
group  meetings.  Often  both  have  Government  and  contractors. 

Working  group  meetings  were  where  lots  of  problems  were  resolved.  Sometimes 
all  the  necessary  functional  specialties  were  present.  However,  most  only  contained  one 
or  two  specialties,  and  other  functional  types  were  ignored.  Often  there  were  both 
Government  and  contractor  personnel  in  these  groups. 
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2.  Proper  Staffing  of  IPT? 

Because  this  was  a  major  program  for  the  developer,  most  “key  skills”  essential  to 
the  project  were  available.  Some  key  skills  were  not  on  the  team  itself,  but  had  to  be 
requested  when  needed.  Some  people  were  split  across  too  many  different  tasks  &  teams. 
This  was  limited  to  the  microwave  electronics  hybrids  design  group  and  the  printed 
circuit  layout  design  people.  Those  were  both  functional  groups,  and  the  TADS/PNVS 
group  didn’t  have  enough  work  to  justify  keeping  them  on  their  team.  But  they  had  as 
good  of  a  priority  with  these  groups  as  any  other  project  in  the  company  in  Orlando. 

There  was  some  personnel  turnover,  which  can  disrupt  the  schedule  of  the  team. 
Many  people  continued  on  the  project  through  pre-production  planning  and  testing. 

The  developer  team  leader  had  high  technical  competence.  He  had  excellent 
design  experience;  however,  his  production  experience  was  mostly  on  smaller  systems. 

3.  Design  to  Manufacturing  Linkage? 

The  developer  had  a  good  relationship  with  suppliers,  production,  design,  and 
upper  management.  Designers  asked  suppliers  for  their  comments  and  suggestions. 
Occasionally,  they  passed  around  the  models  to  the  suppliers  for  their  comments. 

Getting  feedback  from  suppliers  often  has  a  good  affect  on  buyer-supplier 
relations.  Instead  of  being  just  a  customer,  the  supplier  sees  the  buyer  as  somebody  who 
produces  a  useful  product.  The  product  has  value,  and  therefore  manufacturing  and 
delivering  the  supplies  needed  to  make  it,  also  has  value.  Additionally,  the  feedback  can 
generate  improvements  in  use  of  the  supplied  parts,  or  in  manufacture  of  those  parts. 

Production  processes  are  very  important.  Design  engineers  went  to  the  shop  floor 
many  times  to  discuss  them  with  manufacturing  specialists.  The  team  members  met  with 
production  on  the  shop  floor  during  the  latter  part  of  the  development  phase  and  during 
the  transition  to  production  phase.  The  production  representatives  participated  regularly 
in  the  early  and  latter  parts  for  the  development  phase. 
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Though  the  manufacturing  engineers  in  the  production  group  reviewed 
engineering  drawings,  they  did  not  use  computational  models  or  analytic  tools. 
Computer  tools  were  not  yet  available,  nor  were  there  any  of  the  manufacturing  tools 
which  exist  today. 

During  joint  discussions,  they  passed  around  prototypes  of  smaller  components. 
This  facilitated  understanding  and  stimulated  discussion.  A  few  times,  they  held 
planning  meetings  with  both  design  &  production  people.  The  design  team  and  technical 
professionals  from  production  held  unscheduled  and  informal  joint  conversations  about 
the  project  throughout  the  development  phase. 

Occasionally,  they  had  test  articles  or  pre-production  parts  to  discuss  and  examine 
jointly.  Prototypes  and  parts  were  used  in  joint  discussions  late  in  the  development  phase 
and  in  the  transition  to  production  phase. 

4.  Design  to  Supplier  Linkage? 

During  system  development,  the  design  engineers  and  suppliers  worked  closely 
together.  During  joint  discussions,  they  frequently  had  test  articles  or  pre-production 
parts  to  discuss  and  examine  jointly.  The  suppliers  modified  their  hardware  for  this 
specific  job  to  satisfy  the  developer.  They  invited  suppliers  to  planning  meetings  a  few 
times.  However,  this  teamwork  did  not  extend  to  using  computational  models  or  analytic 
tools. 

The  design  team  and  technical  professionals  from  suppliers  had  unscheduled  and 
informal  joint  conversations  about  the  project  during  the  selection  phase  and  all  though 
the  development  phase.  Prototypes  and  parts  were  used  in  joint  discussions  during  the 
latter  development  phase  and  the  transition  to  production.  Significant  effort  was  needed 
to  overcome  suppliers’  not  meeting  delivery  commitments. 


D.  CONCLUSIONS  ON  INTEGRATED  PRODUCT  TEAMS 

This  early  usage  of  teams,  similar  to  IPT’s,  was  fairly  successful.  The  teams 
often  did  not  work  as  well  as  they  could  have  because  there  was  no  firm  policy  to  have  all 
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needed  disciplines  present.  However,  because  the  program  was  large  and  important  for 
the  developer,  they  got  most  of  the  people  and  equipment  that  they  needed.  This  section 
draws  conclusions  concerning  the  maturity  of  the  critical  technologies. 

1.  IPT  Approach  Used? 

TADS/PNVS  employed  multidisciplinary  teams,  also  known  as  cross-functional 
groups.  Because  these  groups  were  not  formally  chartered,  if  an  organization  did  not 
send  anyone  -  due  to  lack  of  manpower  or  travel  funding  -  then  that  group’s  views  and 
expertise  might  not  be  included. 

A  change  in  membership  can  cripple  the  effectiveness  of  an  IPT.  The  synergism 
that  comes  from  working  with  known  people  over  time  is  lost  -  people  are  not 
interchangeable  parts.  Varying  direction  from  an  organization  can  also  cause  all  their 
directions  to  be  ignored.  This  often  happens  in  IPTs  today. 

The  successful  use  of  multidisciplinary  teams  on  this  and  other  commercial  and 
DoD  programs  led  to  the  large-scale  adoption  of  Integrated  Product  Teams. 
Multidisciplinary  teams  were  used  on  the  more  important  projects.  Because  they  were 
successful,  they  began  to  be  used  on  more  and  more  projects.  Additionally,  the  close 
proximity  of  the  team  also  fostered  informal  multidisciplinary  effort.  It  also  engendered 
camaraderie  and  group  cohesion. 

The  TADS/PNVS  developer  had  strong  leadership.  Usually  the  group  was  able  to 
resolve  differences  of  opinion,  but  occasionally  upper  management  had  to  get  involved. 
Program  reviews  were  fairly  good  at  eliciting  problems,  and  the  channels  to  upper  Army 
management  were  quite  effective. 

Pre-IPT  groups  addressed  a  variety  of  problems,  large  and  small,  with  a  fair 
amount  of  success.  Groups  differed  in  the  formality  of  their  organization,  in  the  level  of 
analysis  they  required,  and  whether  they  include  both  Government  and  contractors.  Since 
there  was  no  fonnal  structure,  some  groups  were  only  able  to  handle  smaller  problems, 
while  others  handled  larger  problems  with  considerable  success. 
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2.  Proper  Staffing  of  IPT? 

Most  of  the  necessary  skills  needed  for  the  project  were  actually  on  the  project 
team,  and  a  few  other  skills  were  provided  from  outside  with  a  sufficiently  high  priority. 
Many  of  the  personnel  continued  working  on  the  program  throughout  the  development 
and  testing  period,  though  there  was  some  turnover.  The  developer  team  leader  had 
excellent  technical  development  skills,  though  his  production  experience  was  mostly  on 
smaller  systems. 

3.  Design  to  Manufacturing  Linkage? 

The  developer  worked  and  communicated  well  with  internal  groups  (production, 
design,  and  upper  management)  as  well  as  suppliers.  They  involved  suppliers  in  the 
design  process  to  good  effect.  Occasionally,  smaller  prototype  components,  assemblies, 
test  articles  or  pre-production  parts  were  passed  around  to  facilitate  understanding. 
Production  representatives  participated  in  the  design  process;  and  production  and  design 
groups  met  many  times  to  discuss  production  processes.  Manufacturing  engineers  also 
reviewed  engineering  drawings;  the  more  automated  verification  techniques  that  are 
available  today  didn't  exist  then. 

4.  Design  to  Supplier  Linkage? 

The  developer  and  its  suppliers  worked  fairly  well  together,  on  either  a  formal  or 
an  informal  basis,  examining  prototype  parts  together  and  participating  in  joint  planning. 
However,  they  didn’t  use  computational  models  or  analytic  tools,  which  were  only  just 
becoming  available.  And  significant  effort  was  needed  for  suppliers  to  overcome 
problems  in  meeting  delivery  commitments. 


In  summary,  the  multidisciplinary  teams  eventually  evolved  into  integrated 
product  teams.  These  teams  were  useful  TADS/PNVS  development  tools.  These  teams 
possibly  were  not  as  effective  as  a  formal  IPT,  but  by  integrating  many  disciplines,  they 
were  able  to  solve  many  complex  problems. 
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E.  RECOMMENDATIONS  FOR  FURTHER  STUDY 

1.  IPT  Representation: 

Study  past  and  present  IPTs,  or  groups  not  formally  chartered,  to  detennine  how 
representation  of  functional  disciplines  affects  the  IPTs  effectiveness.  If  an  organization 
didn’t  send  anyone  to  IPT  meetings  -  due  to  lack  of  manpower  or  travel  funding  -  then 
that  group’s  views  and  expertise  might  not  be  included. 

2.  Varying  Membership: 

Study  effectiveness  of  past  and  present  IPTs,  or  groups  not  formally  chartered. 
Some  organizations  send  a  different  person  to  represent  them  to  each  meeting.  Changes 
in  membership  can  cripple  the  effectiveness  of  an  IPT.  Varying  instructions  from  an 
organization  can  also  cause  all  their  directions  to  be  ignored. 

3.  Prime  Contractor  /  Subcontractor  Cooperation: 

Study  how  Prime  and  subcontractors  are  exchanging  more  information  on  their 
process  and  products.  Compare  the  efficiency  of  the  design  process  in  terms  of  problems 
found  early  versus  late  in  the  process.  Also,  compare  this  to  the  Japanese  lean 
manufacturing  model  for  supplier  relations. 

4.  Manufacturing  Processes: 

Study  how  prime  contractors  and  subcontractors  employ  mass  production 

techniques,  craft  production  techniques,  and  lean  production  techniques  across  industries 
and  technologies. 
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VI.  KEY  PROGRAM  MANAGER  ISSUE 


A.  RESEARCH  QUESTION 

This  chapter  answers  the  secondary  research  question,  “What  was  the  key  issue 
that  the  PM  had  to  deal  with  during  the  project  and  how  was  it  dealt  with?”  I  will 
introduce  the  data,  then  analyze  it,  and  finally  draw  conclusions. 


FIGURE  3 

VIDEO  INTERFACE  BETWEEN  TADS/PNVS,  SYMBOL  GENERATOR  &  1HADSS 

Figure  6.  AH-64A  Video  Interfaces,  from  TADS/PNVS  Interfaces  with  other  Mission 
Equipment  of  the  AH-64  Helicopter,  Martin  Marietta  Aerospace  International,  May  1983 


B.  DATA 

1.  Key  Issue  for  PM? 

This  contains  survey  question  12,  only. 

12.  What  was  the  most  difficult  problem  the  Project  Manager  faced,  how  was  the 
problem  dealt  with,  and  what  was  the  impact  of  the  problem  on  the  project  outcome? 
This  quote  is  from  the  government  responder. 
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“The  biggest  problem  was  the  cost  overruns  and  the  underlying  reasons  for  these 
overruns  in  development.  The  source  of  these  cost  overruns  was  due  to  a  couple  of 
factors.  The  primary  problem  was  probably  associated  with  the  acquisition  approach. 
After  the  fly-off  there  was  a  down  select  to  the  winning  contractor.  That  down  select  was 
made  in  the  form  of  the  contract  award  for  the  maturity  phase  of  the  development 
program. 

Each  contractor  submitted  a  proposal  for  the  maturity  phase  as  part  of  the  down- 
select  competition.  It  certainly  was  not  in  the  contractor ’s  best  interest  at  that  stage  to 
cost  the  program  in  their  proposals  to  fully  cover  all  risk  areas.  First,  the  contractor 
would  not  go  out  of  his  way  to  highlight  areas  of  risk  that  the  Government  team  had  not 
identified  and  secondly,  for  those  areas  of  risk  that  were  identified  the  contractors  did  not 
want  to  indicate  the  full  cost  of  the  risk  thereby  putting  themselves  at  a  substantial 
competitive  disadvantage  for  the  down-select  (both  by  proposing  a  higher  cost  than  their 
competitor  and  by  highlighting  greater  technical  risk  with  their  designs  that  required 
greater  funding  in  the  maturity  phase  to  correct). 

Having  said  all  of  this,  I  do  not  feel  that  the  acquisition  approach  was  necessarily 
the  wrong  one  because  the  approach  with  the  competitive  fly-off  and  a  subsequent 
maturity  phase  is  designed  to  significantly  reduce  the  risk  that  the  developed  systems  will 
not  meet  performance  goals. 

Since  the  TADS/PNVS  was  the  number  one  riskiest  technology  in  the  AAH 
program,  this  was  an  appropriate  approach,  and  one  that  was  ultimately  successful.  I 
think  that  both  the  Government  team  and  the  contractor  underestimated  the  effort  it 
would  take  to  implement  the  maturity  phase  design  changes,  meet  all  performance  goals, 
develop  test  equipment,  and  transition  to  production.  The  project  manager,  therefore, 
had  to  deal  with  all  the  issues  that  came  up  because  there  was  more  effort  required  to  get 
the  job  done  in  the  required  time  frame  than  had  been  planned  for.  The  solutions  to 
almost  all  problems  resulted  in  increased  cost.  ” 


C.  ANALYSIS  OF  KEY  PROGRAM  MANAGER  ISSUE 

What  was  the  key  issue  that  the  PM  had  to  deal  with  during  the  project  and  how 
was  it  dealt  with?  The  key  issue  was  cost  overruns,  which  were  due  to  several  factors. 
The  developers  needed  to  win  a  competitive  contract  based  on  cost.  And  if  a  contractor 
explored  and  expounded  the  risks,  their  cost  would  realistically  be  higher  than  their 
competitors.  Please  note  that  Chapter  IV,  User  Support  and  Funding  Stability,  also 
relates  to  funding. 


Cost  overruns  and  lowest  bidder  who  hides  risks: 
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Developers  downplayed  risks  brought  up  by  the  Government,  because  by  fully 
expounding  the  risks  of  their  design,  the  developer  would  have  shown  that  their  proposal 
was  under-funded,  so  risks  were  not  really  explored  or  mentioned.  This  scoring  of  risks 
was  held  against  the  contractor's  design,  rather  than  recognizing  that  the  risks  are  inherent 
in  the  Government's  project  requirements.  This  may  have  been  the  best  contract  vehicle 
at  the  time,  but  it  does  tend  to  reward  the  hiding  of  information. 

It  was  in  the  interests  of  neither  the  Government  PMO  nor  the  Prime  contractor, 
to  have  a  reasonable  program  cost  at  the  beginning  of  the  contract.  The  Prime  wanted  to 
win  the  contract  in  a  competitive  environment.  The  Government  PMO  was  trying  to  get 
the  best  value  for  the  Government. 

Reprogramming  funds  for  a  program  is  expensive.  Besides  the  schedule  loss 
while  you  are  going  through  the  effort,  there  is  the  schedule  loss  due  to  going  back  and 
doing  risk  reduction  you  should  have  done  earlier,  acquiring  parts/equipment/facilities  on 
short  notice,  and  also  redesigning  the  system.  Each  of  these  four  activities  has  an 
associated  cost.  Additionally,  there  is  the  cost  of  materials  acquired  but  no  longer 
needed.  Doing  all  these  cost  and  schedule  activities  later  in  the  program  always  costs 
more  than  if  they  were  on  the  program  schedule  from  day  one. 

Even  though  the  PMO  ‘knew’  that  the  program  probably  could  not  succeed  at  the 
initial  cost,  and  that  the  Government  would  have  to  provide  more  money,  the  strategy 
was  that  the  profit  to  the  contractor  was  based  on  the  initial  program  cost.  It  is  arguable 
whether  this  savings  in  profit  to  the  contractor  was  offset  by  the  cost  of  the  inefficiency 
of  the  total  program  turbulence  and  review  resulting  from  cost  overruns  and 
reprogramming  additional  funds.  Sadly,  cost  overruns  on  the  TADS/PNVS  were  a 
common  occurrence,  given  the  way  the  financial  system  was  set  up. 

Although  making  contract  decisions  based  entirely  upon  cost  was  common  at  the 
time,  today  more  contracting  decisions  are  based  upon  a  variety  of  other  factors, 
including  technical  parameters. 

The  TADS  /  PNVS  was  recognized  as  the  number  one  riskiest  technology  on  the 
AAH  program,  and  the  system  was  ultimately  successful,  though  at  double  the  original 
contract  price.  However,  significant  process  improvements  are  possible  in  the 
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development  and  contracting  strategy.  For  example,  development  contracts  are  routinely 
based  on  more  than  just  cost.  Reporting  risks  can  be  scored  as  value-added  infonnation. 
And  these  risks  can  be  used  to  evaluate  all  contracts,  not  just  the  contractor  who 
mentioned  them  -  although  they  may  not  be  inherent  in  other  contractors'  designs. 

The  Government  and  developers  should  enter  teaming  relationships  early  to 
identify  risk  areas,  and  the  developers  should  be  rewarded  for  this  value-added  activity. 
Finding  technical  risks  later  in  the  program  is  a  common  occurrence,  but  it  should  be 
minimized  as  much  as  possible.  Some  programs  are  cancelled  because  technical  risks 
grow  beyond  the  end  worth  of  the  system.  Finding  risks  earlier  saves  money,  because 
fixing  something  is  always  less  costly  at  the  beginning  of  a  program.  Going  back  to  the 
Government  for  more  funds,  or  to  the  PM,  to  higher  headquarters,  or  to  Congress,  could 
be  a  decision  point  for  canceling  the  program. 


The  Prime  and  Subcontracts: 

TADS/PNVS  Development  contract  was  a  Prime  contract,  not  a  subcontract  to 
Hughes  Helicopters.  Both  the  TADS/PNVS  and  the  Hughes  Apache  contract  had  clauses 
in  them  for  an  Integration  and  Configuration  Working  Group.  Integration  is  a  potential 
problem. 

Having  direct  contract  with  a  developer  of  a  subsystem  has  advantages  and 
disadvantages.  It  gives  the  Government  more  control  to  have  a  direct  contract,  more 
control  over  their  development  processes,  and  over  the  contract  type.  TADS/PNVS  was 
the  highest  risk  item  on  the  Apache  development  program,  and  warranted  a  separate 
project  office.  Having  this  separate  office,  and  a  separate  prime  contract  is  more  work, 
but  it  increases  the  Government's  ability  to  control  the  risk. 

But  then  there  are  questions  concerning  how  you  integrate  the  subsystem  into  the 
prime  contractor’s  system  or  vehicle.  You  can  put  a  clause  in  the  prime's  contract  that 
they  must  integrate  the  subsystem,  and  work  with  the  other  contractor  to  do  so,  but  there 
are  still  some  liability  issues  that  may  arise.  If  redesign  is  necessary  in  order  to  interface 
the  subsystem,  then  the  Government  could  be  liable  for  the  cost.  The  solution  that  the 
TADS/PNVS  program  chose  handled  this  problem  very  well. 
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Primes  still  use  fixed-cost  contracts  for  subcontracts  today.  The  Government  has 
sworn  off  fixed-cost  contracts,  because  development  is  too  high  risk.  However, 
subcontracts  are  frequently  where  the  highest  risk  parts  of  the  program  lie.  Often  the 
prime's  share  of  the  work,  wiring  the  vehicle,  and  installing  the  components,  is  a  much 
lower  risk  activity. 

Cost  Probabilities: 

The  probable  program  cost  at  the  start,  based  on  risk  and  cost  drivers,  can  be 
graphed  -  see  the  figure  below.  It  would  probably  be  some  type  of  bell-shaped  curve, 
with  maximum  probability  in  the  middle.  The  four  vertical  lines  represent  the  1st,  5th, 
50th,  and  99th  percentile  probabilities  of  the  program  cost.  Starting  off  the  program  with  a 
budget  at  the  5th  percentile  probability  is  quite  risky. 


Contract  Costs:  Bell-Shaped  Curve 


Figure  7.  Contract  Costs  Graph 

One  problem  with  setting  the  contract  price  at  the  5th  percentile  probability  point 
is  that  it  forces  the  program  to  “restructure,”  “realign,”  or  “reprogram”  too  often.  This 
effort  is  a  financial  drain  on  the  program’s  cost  and  schedule,  as  well  as  an 
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embarrassment  to  all  concerned.  And  of  course,  little  program  work  is  done  while  you 
are  involved  in  the  meetings  and  discussions  of  the  restructuring,  though  you  are  ’burning’ 
cost  and  schedule. 

Starting  off  at  the  5  th  percentile  probability  level  also  causes  some  risky  behavior. 
If  you  had  more  money,  you  could  undertake  some  risk-diminishing  strategies  earlier  in 
the  program,  and  possibly  avoid  some  schedule  slips.  Doing  risk  reduction  later  on  costs 
more,  because  it  may  invalidate  design  decisions,  and  all  later  design  and  development 
work  based  on  those  decisions.  At  the  lower  budget  cost,  you  are  required  to  allocate 
almost  all  your  resources  on  well-understood  requirements,  which  generally  have 
predictable  costs,  and  very  little  for  risk  reduction  efforts. 

It  is  better  to  start  the  program  off  with  a  budget  nearer  the  50th  percentile 
probability.  This  will  allow  funding  risk  reduction  studies  and  an  emergency  funding 
reserve.  Just  as  the  AAH  PMO  had  a  financial  reserve,  the  contractor  PMO  will  establish 
a  financial  reserve  within  his  own  budget. 


D.  CONCLUSIONS 

Cost  overruns  are  a  large  concern  in  Government  programs.  TADS  /  PNVS  cost 
concerns  stemmed  from  hiding  or  downplaying  risks  even  before  the  contractor  won  the 
contract. 

Playing  the  game  to  get  a  contract  started  for  less  than  the  contractor  needs  is 
risky  -  cost  overruns  are  likely.  Most  companies  that  have  sued  the  Government  to 
recover  cost  overruns  have  succeeded.  Realistic  contract  prices  at  the  beginning  are  more 
cost  effective,  avoiding  costly  reprogramming/restructuring,  the  political  turmoil  inherent 
in  asking  congress  for  more  money,  and  the  risk  of  contract  disputes  in  court. 

Programs  are  very  rarely  cancelled  for  cost  considerations.  Once  a  program  is 
started  and  gains  its  military,  congressional,  and  defense  contractor  adherents,  it  is 
difficult  to  cancel.  Although  making  contract  decisions  based  entirely  upon  cost  was 
common  at  the  time,  today  more  contracting  decisions  are  based  on  a  variety  of  other 
factors,  including  technical  parameters. 
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Best  Value  is  not  always  guaranteed  by  choosing  the  lowest  bidder  on  a  contract. 
Program  managers  need  to  reduce  risks,  if  their  program  is  to  succeed.  One  possibility  is 
that  the  request  for  proposal  could  request  that  the  technical  risks  inherent  in  the  project 
requirements  be  listed  separately  by  a  contractor,  and  compared  to  other  contractors’ 
proposals.  This  could  be  scored  in  favor  of  the  contractor  -  they  should  get  the  contract 
because  they  understand  the  problem  better.  Of  course,  they  should  also  propose  how  to 
deal  with  these  risks. 

Prime  contracts  and  subcontracts  should  be  considered  for  high-risk  subsystems. 
If  a  prime  contract  is  used,  it  adds  workload  to  the  Government,  but  can  lower  total 
program  risks  if  effectively  managed.  Integration  is  a  bigger  issue  with  multiple  prime 
contracts  on  one  program.  If  the  work  is  subcontracted,  the  Government  should  consider 
what  type  of  the  contract  the  prime  will  use,  and  realize  they  will  have  less  control  over 
development. 

If  most  programs  go  over  cost,  then  the  system  is  probably  too  risky.  It  is  better 
to  predict  the  cost  realistically,  and  commit  the  appropriate  amount.  Some  budget  people 
advocate  squeezing  a  program  a  bit  to  encourage  cost  reduction  efforts,  but  cutting  back 
by  50%  is  not  realistic.  The  5th  percentile  of  the  probable  cost  is  too  low.  Planners 
should  try  to  target  the  40th  to  50th  percentile  of  the  probable  cost.  But  even  then,  some 
programs  will  go  over  the  contract  price,  and  a  program  manager  or  his  PEO  could  have  a 
contingency  fund.  Starting  with  a  reasonable  contract  price  allows  more  realistic 
planning  and  earlier  risk  reduction  efforts. 


E.  RECOMMENDATIONS  FOR  FURTHER  STUDY 
1.  Financial  Management  and  Contract  Types: 

Study  how  contract  types  have  changed  over  the  years  from  1980  to  the  present, 
and  the  policies  for  use  of  these  contract  types.  Study  how  program  financial 
management  varies  with  contract  type,  and  what  affect  this  has  on  the  program  manager's 
ability  to  control  various  aspects  of  the  program. 
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2. 


Prime  and  Subcontracts: 


Study  how  some  programs  are  run  with  only  one  prime  contract,  and  a  variety  of 
subcontracts,  and  other  have  many  prime  contracts.  Study  ease  of  integration  and 
management  of  development  risks. 

3.  Study  Program  Risk  Reduction  Proposals  in  Contracts: 

Study  how  various  DoD  contracts  have  proposed  reducing  risks,  and  how 
successful  these  programs  have  been  in  reducing  risks. 


56 


VII.  SIMULATION  AND  TESTING  STRATEGY 


A.  PRIMARY  RESEARCH  QUESTION 

This  chapter  answers  the  primary  research  question,  “What  was  the  simulation 
and  testing  strategy  for  the  system,  and  did  that  strategy  adequately  evaluate  the  system 
for  its  ultimate  operational  use?”  I  will  introduce  the  data,  then  analyze  it,  and  finally 
draw  conclusions. 


EOB-Electro-Optical  Bench  Host  Shelter 

UUT-Unit  Under  Test 

+302-31 


Figure  11-1.  AH-64D  Automatic  Test  Station  Interface 

Figure  8.  TADS  Automatic  Test  Station  Interface,  from  Interface  Control  Document  for  the 
Longbow  Apache  Target  Acquisition  Designation  System  PNVS  (TADS  PNVS),  26 

April  1999 
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B. 


DATA 


In  addition  to  data  from  the  combined  survey,  I  included  some  related  information 
from  a  General  Accounting  Office  (GAO)  study  on  the  DoD  testing  process.  This  study 
contrasts  the  test  processes  during  product  development,  between  civilian  company 
testing  and  DoD  testing,  and  shows  how  DoD  testing  can  be  improved. 


1.  Test  Approach  Used? 

This  covers  survey  questions  VI  through  VI 5,  Validation  Activities. 

Validation  Activities:  Testing  and  Simulation 

V 1 .  Was  a  failure  modes  and  effects  analysis  done  on  the  system?  Yes 
Via.  If  yes,  was  it  used  to  help  establish  the  test  plan?  Yes 

This  analysis  drove  the  test  requirements  for  production  test  equipment  and  for 
fielded  automatic  test  equipment. 

This  analysis  drove  the  test  requirements  for  production  test  equipment  and  for 
fielded  automatic  test  equipment.  The  FMECA  looks  for  things  likely  to  break,  and  its 
results  influenced  qualification  tests  and  simulation.  Performance  tests  were  run  under 
extreme  vibration  conditions.  Production  planning  was  done  during  development,  as  a 
logistics  effort  to  identify  support  requirements.  They  used  the  same  test  equipment  in 
production  as  at  the  Aviation  Intermediate  Maintenance  (AVIM)  facility. 


For  individual  components: 

Questions  V2  through  V6  were  answered  using  these  possible  answers.  More  than 


one  answer  is  permitted. 


1 

2 

3 

4 

5 

9 

Prime 

Suppliers 

Anny 

center/lab 

Other 

Government  org. 

Not  done 
on  project 

Don’t 

know 

V2.  Was  there  testing  to  see  if  the  individual  components  of  the  system 
worked?  What  organization(s)  did  this  testing?  Prime  and  Suppliers 

V3.  Were  there  simulations  run  to  see  if  the  individual  components  of  the 
system  worked?  What  organization(s)did  these  simulations?  Prime  and  Suppliers 

For  integrated  components  in  controlled  setting: 

V4.  Were  the  components  tested  working  together  in  a  controlled  setting? 

What  organization(s)  did  this  testing?  Prime,  Suppliers,  and  Other  Government 
organization 

V5.  Were  there  simulations  of  the  components  working  together  in  a 
controlled  setting?  What  organization(s)  did  this?  Prime  and  Army  center/lab 
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For  integrated  components  in  a  realistic  setting: 

V6.  Was  there  testing  of  the  components  working  together  in  a  realistic 
setting?  What  organization(s)  did  this  testing?  Prime  and  Army  center/lab 

V7.  Was  a  hardware-in-the-loop  type  systems  integration  simulation  laboratory 

used? 

V7a.  To  see  if  the  individual  components  of  the  system  worked:  Yes. 
V7b.  To  see  if  integrated  components  worked  in  controlled  setting:  Yes. 
TADS/PNVS  components  were  tested  in  the  completed  system  using  “ aircraft 
simulator”  test  equipment.  The  TADS/PNVS  itself  was  testing  in  the  aircraft 
manufacturer ’s  Systems  Integration  Laboratory. 

V8.  Recalling  the  total  effort  (100%)  spent  on  testing  and  simulations,  please 
allocate  the  percent  of  that  total  that  were: 

15  %  spent  to  see  if  the  individual  components  of  the  system  worked 
5  %  spent  to  see  if  integrated  components  worked  in  controlled  setting 
80  %  spent  to  see  if  integrated  components  worked  in  a  realistic  setting 
m  %  Spent  on  any  other  validation  purpose 
100  % 


Questions  V9  through  V15  were  answered  using  these  possib 


e  answers: 


1 

2 

3 

4 

5 

9 

Strongly 

disagree 

Disagree 

somewhat 

Neither  agree 
nor  disagree 

Agree 

somewhat 

Strongly 

agree 

Don’t 

know 

V9.  Knowledge  from  validation  work  was  used  consistently  to  improve 
components  and  system.  Strongly  agree. 

V10.  Project  test  philosophy  was  to  “Break  it  big  early.”  Agree  somewhat. 

V 1 1 .  Component  and  system  maturity  were  validated  at  the  right  times  in  the 
program.  Strongly  agree. 

V12.  The  project  and  the  testing  community  had  an  adversarial  relationship. 
Strongly  disagree. 

V13.  Most  project  validation  events  produced  quality  results.  Agree  somewhat. 
V14.  The  project  didn’t  recognize  important  lessons  that  validation  work 
uncovered.  Strongly  disagree. 

VI 5.  Sometimes  the  project  settled  for  less  than  the  best  validation  method. 
Strongly  disagree. 


2.  GAO  Research: 

This  data  is  excerpts  from:  General  Accounting  Office  (GAO)  Report,  Best 
Practices,  A  More  Constructive  Test  Approach  Is  Key  to  Better  Weapon  System 
Outcomes,  GAO/NSIAD-OO-199,  July  2000. 
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Report  Abstract:  This  report  examines  (1)  how  the  conduct  of  testing  and 
evaluation  affects  commercial  and  Defense  Department  (DOD)  program 
outcomes,  (2)  how  best  commercial  testing  and  evaluation  practices 
compare  with  DOD's,  and  (3)  what  factors  account  for  the  differences  in 
these  practices.  GAO  found  that  commercial  firms  use  testing  to  expose 
problems  earlier  than  the  DOD  programs  GAO  visited.  Commercial  linns’ 
testing  and  evaluation  validates  products'  maturity  based  on  three  levels  at 
specific  points  in  time,  which  works  to  preclude  "late-cycle  churn"  or  the 
scramble  to  fix  a  significant  problem  discovered  late  in  development. 
Late-cycle  churn  has  been  a  fairly  common  occurrence  on  DOD  weapon 
systems,  where  tests  of  a  full  system  identify  problems  that  often  could 
have  been  found  earlier.  DOD's  response  to  such  test  results  typically  is  to 
expend  more  time  and  money  to  solve  the  problems— only  rarely  are 
programs  terminated.  The  differences  in  testing  practices  reflect  the 
different  demands  commercial  firms  and  DOD  impose  on  program 
managers.  Leading  commercial  firms  insist  that  a  product  satisfy  the 
customer  and  make  a  profit.  Success  is  threatened  if  unknowns  about  a 
product  are  not  resolved  early  when  costs  are  low  and  more  options  are 
available.  Testing  is  constructive  and  eliminates  unknowns.  Success  for  a 
weapons  system  is  centered  on  providing  a  superior  capability  within 
perceived  time  and  funding  limits.  Testing  plays  a  less  constructive  role, 
because  test  results  often  become  directly  linked  to  funding  and  other  key 
decisions  and  can  jeopardize  program  support.  Such  a  role  creates  a  more 
adversarial  relationship  between  testers  and  program  managers. 


Purpose  Despite  good  intentions  and  some  progress  by  the  Department  of 
Defense  (DOD),  weapon  system  programs  still  suffer  from  persistent 
problems  associated  with  late  or  incomplete  testing.  Often,  the  fate  of  a 
program  is  jeopardized  by  unexpectedly  poor  test  results.  In  such  cases, 
testing  becomes  a  watershed  event  that  attracts  unwanted  attention  from 
decision  makers  and  critics.  The  discovery  of  problems  in  complex 
products  is  a  nonnal  part  of  any  development  process,  and  testing  is 
perhaps  the  most  effective  tool  for  discovering  such  problems.  However, 
why  surprises  in  testing  repeatedly  occur  and  why  such  results  polarize 
organizations  into  proponents  and  critics  of  programs  have  proven  elusive 
questions  to  answer.  Indeed,  numerous  solutions  proposed  over  the  years 
by  different  DOD  leaders  and  distinguished  outside  panels  have  not  had 
much  effect. 


Lessons  learned  by  leading  commercial  firms  in  developing  new  products 
are  applicable  to  the  management  and  testing  of  weapon  systems.  These 
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firms  achieve  the  type  of  outcomes  DOD  seeks:  they  develop  more 
sophisticated  products  faster  and  less  expensively  than  their  predecessors. 
Commercial  firms  have  found  constructive  ways  of  conducting  testing  and 
evaluation  that  help  them  avoid  being  surprised  by  problems  late  in  a 
product’s  development.  In  response  to  a  request  from  the  Chairman  and 
the  Ranking  Minority  Member,  Subcommittee  on  Readiness  and 
Management  Support,  Senate  Committee  on  Armed  Services,  GAO 
examined  (1)  how  the  conduct  of  testing  and  evaluation  affects 
commercial  and  DOD  program  outcomes,  (2)  how  best  commercial  testing 
and  evaluation  practices  compare  with  DOD’s,  and  (3)  what  factors 
account  for  the  differences  in  these  practices. 


Background  The  fundamental  purpose  of  testing  and  evaluation  does  not 
differ  for  military  and  commercial  products.  Testing  is  the  main 
instrument  used  to  gauge  the  progress  being  made  when  an  idea  or  concept 
is  translated  into  an  actual  product....  In  both  DOD  and  commercial  firms, 
product  testing  is  conducted  by  organizations  separate  from  those 
responsible  for  managing  product  development. 

Results  in  Brief  For  the  leading  commercial  firms  GAO  visited,  the  proof 
of  testing  and  evaluation  lies  in  whether  a  product  experiences  what  one 
firm  called  “late-cycle  churn,”  or  the  scramble  to  fix  a  significant  problem 
discovered  late  in  development. . . . 

On  the  weapon  programs,  system-level  testing  carried  a  greater  share  of 
the  burden.  Earlier  tests  were  delayed,  skipped,  or  not  conducted  in  a  way 
that  advanced  knowledge....  Leading  commercial  firms  have  learned  to 
insist  that  a  product  satisfy  the  customer  and  make  a  profit.  Success  is 
threatened  if  managers  are  unduly  optimistic  or  if  unknowns  about  a 
product  are  not  resolved  early,  when  costs  are  low  and  more  options  are 
available.  The  role  of  testing  under  these  circumstances  is  constructive,  for 
it  helps  eliminate  unknowns.  Product  managers  view  testers  and  realistic 
test  plans  as  contributing  to  a  product’s  success.  Success  for  a  weapon 
system  program  is  different;  it  centers  on  attempting  to  provide  a  superior 
capability  within  perceived  time  and  funding  limits.  Success  is  influenced 
by  the  competition  for  funding  and  the  quest  for  top  performance; 
delivering  the  product  late  and  over  cost  does  not  necessarily  threaten 
success.  Testing  plays  a  less  constructive  role  in  DOD  because  a  failure  in 
a  key  test  can  jeopardize  program  support.  Specifically,  test  results  often 
become  directly  linked  to  funding  and  other  key  decisions  for  programs. 
Such  a  role  creates  a  more  adversarial  relationship  between  testers  and 
program  managers. 
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Principal  Findings 


Problems  Found  Late  in  Development  Signal  Weaknesses  in 
Testing  and  Evaluation 

Over  the  years,  GAO  found  numerous  examples  of  late-cycle  churn  in 
DOD  programs,  regardless  of  their  size,  complexity,  or  product  type.  More 
recent  examples  include  the  following: 

•  The  DarkStar  unmanned  aerial  vehicle  crashed  during  initial  flight  tests. 
DOD  spent  twice  the  planned  money  and  time  to  redesign  and  retest  the 
aircraft,  eventually  terminating  the  program. 

Testing  Early  to  Validate  Product  Knowledge  Is  a  Best  Practice 

Leading  commercial  firms  GAO  visited  think  in  terms  of  validating  that  a 
product  works  as  intended  and  use  testing  and  evaluation  as  a  means  to 
that  end.  To  limit  the  burden  on  the  product’s  third  maturity  level 
(operating  in  a  realistic  environment),  leading  firms  ensure  that  (1)  the 
right  validation  events — tests,  simulations,  and  other  means  for 
demonstrating  product  maturity — occur  at  the  right  times,  (2)  each 
validation  event  produces  quality  results,  and  (3)  the  knowledge  gained 
from  an  event  is  used  to  improve  the  product.  The  firms  hold  challenging 
tests  early  to  expose  weaknesses  in  a  product’s  design.  AT&T  refers  to 
this  as  a  “break  it  big  early”  philosophy. . . . 


Different  Incentives  Make  Testing  a  More  Constructive  Factor 
in  Commercial  Programs  Than  in  Weapon  System  Programs 


...  Test  results  tend  to  become  scorecards  that  demonstrate  whether  the 
program  is  ready  to  proceed  or  to  receive  the  next  increment  of  funding. 
Whereas  testing  and  evaluation  of  commercial  products  mainly  benefits 
the  product  manager,  in  DOD,  testing  and  evaluation  is  more  for  the 
benefit  of  the  testers  and  decision  makers  above  the  program  manager. 
Managers  thus  have  incentives  to  postpone  difficult  tests  and  to  limit  open 
communication  about  test  results.  Managers  in  both  the  DarkStar 


62 


unmanned  aerial  vehicle  and  the  Standoff  Land  Attack  Missile  programs 
also  overruled  testers  because  of  funding  and  schedule  pressures. 


Recommendations  To  lessen  the  dependence  on  testing  late  in 
development  and  to  foster  a  more  constructive  relationship  between 
program  managers  and  testers,  GAO  recommends  that  the  Secretary  of 
Defense  instruct  acquisition  managers  to  structure  test  plans  around  the 
attainment  of  increasing  levels  of  product  maturity,  orchestrate  the  right 
mix  of  tools  to  validate  these  maturity  levels,  and  build  and  resource 
acquisition  strategies  around  this  approach.  GAO  also  recommends  that 
validation  of  lower  levels  of  product  maturity  not  be  deferred  to  the  third 
level.  Finally,  GAO  recommends  that  the  Secretary  require  that  weapon 
systems  demonstrate  a  specified  level  of  product  maturity  before  major 
programmatic  approvals.” 


Chapter  2.  Problems  Found  Late  in  Development  Signal 
Weaknesses  in  Testing  and  Evaluation 

...  Problems  are  most  devastating  when  they  delay  product  delivery, 
increase  product  cost,  or  “escape”  to  the  customer. 

Chapter4:  Different  Incentives  Make  Testing  a  More 
Constructive  Factor  in  Commercial  Programs  Than  in  Weapon 
System  Programs 

The  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and 
Logistics,  before  he  took  office,  pinpointed  the  following  differences  in 
commercial  and  DOD  testing: 

In  the  commercial  world,  the  reason  for  testing  and  evaluating  a  new  item 
is  to  detennine  where  it  will  not  work  and  to  continuously  improve  it ...  . 
Thus  testing  and  evaluation  is  primarily  for  the  purpose  of  making  the  best 
possible  product,  and  making  it  as  robust  as  possible  ....  By  contrast, 
testing  and  evaluation  in  the  Department  of  Defense  has  tended  to  be  a 
final  exam,  or  an  audit,  to  see  if  a  product  works.  Tests  are  not  seen  as  a 
critical  element  in  enhancing  the  development  process;  the  assumption  is 
that  the  product  will  work  and  it  usually  does.  Under  these  conditions,  the 
less  testing  the  better — preferably  none  at  all.  This  rather  perverse  use  of 
testing  causes  huge  cost  and  time  increases  on  the  defense  side,  since  tests 
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are  postponed  until  the  final  exam  and  flaws  are  found  late  rather  than 
early.  1  (1  Defense  Conversion:  Transforming  the  Arsenal  of 

Democracy;  MIT  Press,  1995.) 

Commercial  Incentives  Foster  Candor  and  Realism  in  Product 
Validation 

Once  a  company  decides  to  launch  a  product  development,  strong 
incentives,  grounded  in  the  business  case,  encourage  a  focus  on  product 
validation  to  keep  the  program  on  track.  To  meet  market  demands,  leading 
commercial  companies  plan  around  comparatively  short  cycle  times — 
often  less  than  2  years — to  complete  a  product’s  development.  These  short 
time  frames  make  customer  acceptance  and  return  on  investment  close  at 
hand.  Consequently,  production  looms  as  a  near-tenn  reality  that 
continues  to  influence  subsequent  product  decisions  within  the  framework 
of  the  business  case. 

To  deliver  the  product  on  time,  commercial  firms  insist  on  validating  the 
maturity  of  technologies  before  they  are  allowed  onto  a  new  product. 


Testing  Is  Perceived  as  Impeding  the  Success  of 
Weapon  System  Programs 

The  basic  management  goal  for  a  weapon  system  program  in  DOD  is 
similar  to  that  of  a  commercial  product:  to  develop  and  deliver  a  product 
that  meets  the  customer’s  needs.  However,  the  pressures  of  successfully 
competing  for  the  funds  to  start  and  sustain  a  weapon  system  program 
create  incentives  for  launching  programs  that  embody  more  technical 
unknowns  and  less  knowledge  about  the  performance  and  production  risks 
they  entail.  On  the  basis  of  our  present  and  previous  work,  as  well  as  our 
review  of  outside  studies,  such  as  those  sponsored  by  DOD,  we  have 
identified  several  key  factors  that  affect  the  business  case  for  starting  a 
new  weapon  system  program. 

Annual  funding  is  approved  -> 

User  requirements  exist  -> 

Program  promises  best  capability  -> 

Program  looks  affordable  ->(back  to  beginning) 
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Testing  Can  Pose  a  Serious  Threat  to  a  DOD  Program 

Within  the  DOD  business  case  for  the  programs  we  reviewed,  test  results 
tended  to  become  scorecards  that  demonstrated  to  decision  makers  that  the 
program  was  ready  to  proceed  to  the  next  acquisition  phase  or  to  receive 
the  next  increment  of  funding.  As  a  result,  testing  operated  under  a  penalty 
environment;  if  tests  were  not  passed,  the  program  might  look  less 
attractive  and  be  vulnerable  to  funding  cuts.  Managers  thus  had  incentives 
to  postpone  difficult  tests  and  limit  open  communication  about  test  results. 
Under  these  conditions,  demonstrations  that  show  enough  progress  to 
continue  the  program  are  preferred  over  actual  tests  against  criteria,  which 
can  reveal  shortfalls.  Accordingly,  DOD  testers  are  often  seen  as 
adversaries  to  the  program.  In  general,  testers  are  often  organizationally 
removed  from  the  design  and  development  effort  and  are  viewed  as 
outsiders.  Unlike  their  commercial  counterparts,  they  do  not  have  a  strong 
voice  in  early  program  planning  decisions.  As  a  result,  their  authority  or 
influence  is  limited,  and  they  are  often  overruled  in  decisions  to  proceed 
with  programs  despite  testing  weaknesses. 

The  role  testing  plays  in  DOD  programs  was  analyzed  in  a  September 
1999  report  from  the  Defense  Science  Board. 3  The  Board  concluded  that 
the  “response  to  perceived  test  failures  is  often  inappropriate  and 
counterproductive.”  Instead  of  using  testing,  especially  in  the  early  stages, 
as  a  vital  learning  mechanism  and  an  opportunity  to  expand  product 
knowledge,  testing  is  often  used  as  a  basis  for  withholding  funding,  costly 
rescheduling,  or  threats  of  cancellation. 

DOD  Testing  Impaired  by  Optimism  and  Insufficient  Resources 

Although  DOD  does  extensive  test  and  resource  planning,  the  planning  on 
the  weapon  systems  we  reviewed  was  often  undercut  by  unrealistic 
assumptions.  DOD’s  acquisition  regulation  5000. 2R  requires  fonnal  test 
plans  and  resource  estimates  for  every  weapon  system  program  that  must 
be  reviewed  and  approved  by  numerous  organizations.  This  fonnal 
process  does  not  guarantee  that  the  program  will  comply  with  the  plan  or 
receive  the  resources  requested  or  that  the  plan  itself  is  realistic.  On  the 
programs  we  reviewed,  pressures  to  keep  schedule  and  cost  estimates  as 
low  as  possible  forced  managers  into  optimistic  plans  that  presume 
success  instead  of  anticipating  problems.  Test  resources  and  schedules 
were  assigned  accordingly.  The  resultant  test  plans  eventually  proved 
unexecutable  because  they  underestimated  the  complexity  and  the 
resources  necessary  to  validate  the  product’s  maturity.  Typically,  the  time 
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and  money  allocated  to  testing  was  more  a  by-product  than  a  centerpiece 
of  the  product  development  estimate. 


The  DarkStar  test  approach  had  similar  constraints.  The  contractor 
developed  a  test  plan  that  accommodated  cost  and  schedule  limits,  but  did 
not  address  the  range  of  technical  parameters  that  needed  to  be 
investigated.  Problems  were  noted  during  testing,  but  because  of  schedule 
and  cost  pressures,  minimal  attempts  were  made  to  correct  them.  The 
safety  investigation  board,  which  investigated  after  the  vehicle  crashed, 
reported  that  “scheduling  was  dictated  by  programmatic  pressures  rather 
than  sound  engineering  processes”  and  “the  overriding  driver  repeatedly 
appeared  to  be  schedule  and  budget.”  The  funding  and  schedule 
constraints  were  imposed  without  considering  what  resources  were  needed 
to  adequately  mature  and  integrate  DarkStar  components  into  a  system. 
Ironically,  the  resources  to  redesign  and  retest  the  system — double  the 
original  estimate — were  made  available  only  after  serious  problems 
occurred  under  the  original  plan. 


Conclusions 

For  testing  and  evaluation  to  become  part  of  a  constructive  effort  to 
validate  the  maturity  of  new  weapon  systems  in  DOD,  the  role  it  plays  and 
the  incentives  under  which  it  operates  must  change.  Currently,  testing  and 
testers  are  not  seen  as  helping  the  product  succeed  but  as  potential 
obstacles  for  moving  forward.  They  become  more  closely  linked  with 
funding  and  program  decisions  and  less  likely  to  help  the  weapon  system 
improve.  Given  the  pressures  on  program  managers  to  keep  development 
cost  and  schedule  estimates  low,  being  optimistic  and  reluctant  to  report 
difficulties  is  more  important  to  program  success  than  planning  a  realistic 
validation  effort  to  discover  design  and  other  problems.  Attempts  by 
decision  makers  to  impose  cost  and  schedule  constraints  on  a  program 
without  full  consideration  of  what  is  required  to  reach  product  maturity 
levels  becomes  a  false  discipline  that  can  intensify  pressures  to  defer  or 
weaken  testing,  thereby  increasing  the  potential  for  late  cycle  churn.  If 
DOD  is  successful  in  taking  actions  that  respond  to  our  previous 
recommendations,  especially  those  that  will  reduce  the  pressure  to  oversell 
programs  at  their  start,  the  Department  will  have  taken  a  significant  step 
toward  changing  what  constitutes  success  in  weapon  systems  and  making 
testing  and  evaluation  a  more  constructive  factor  in  achieving  success. 
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Recommendations 


To  lessen  the  dependence  on  testing  late  in  development  and  foster  a  more 
constructive  relationship  between  program  managers  and  testers,  we 
recommend  that  the  Secretary  of  Defense  instruct  the  managers  and  testers 
of  weapon  system  programs  to  work  together  to  define  levels  of  product 
maturity  that  need  to  be  validated,  structure  test  plans  around  reaching 
increasing  levels  of  product  maturity,  and  orchestrate  the  right  mix  of  tools 
to  validate  these  levels.  Acquisition  strategies  should  then  be  built  and 
funded  to  carry  out  this  approach.  Such  a  focus  on  attaining  knowledge, 
represented  by  product  maturity  levels,  can  guard  against  the  pressures  to 
forego  valuable  tests  to  stay  on  schedule  or  to  hold  tests  that  do  not  add 
value  to  the  product.  This  approach,  which  creates  common  ground 
between  testers  and  product  managers  in  leading  commercial  firms 
without  compromising  independence,  still  demands  that  the  product  or 
weapon  system  being  matured  meet  the  needs  of  the  customer. 


We  also  recommend  that  Secretary  of  Defense  not  let  the  validation  of 
lower  levels  of  product  maturity — individual  components  or  systems  in  a 
controlled  setting — be  deferred  to  the  higher  level  of  system  testing  in  a 
realistic  setting.  Although  the  mix  of  testing  and  evaluation  tools  may 
change  and  the  acquisition  strategy  may  be  altered  during  the  course  of  a 
development,  the  focus  on  attaining  product  maturity  levels  should  not 
change.  This  discipline  should  also  help  guard  against  the  practice  of 
setting  cost  and  schedule  constraints  for  programs  without  considering  the 
time  and  money  it  takes  to  sensibly  validate  maturity.  Finally,  we 
recommend  that  the  Secretary  of  Defense  require  weapon  systems  to 
demonstrate  a  specified  level  of  product  maturity  before  major 
programmatic  approvals.  In  doing  so,  the  Secretary  may  also  need  to 
establish  interim  indicators  of  product  maturity  to  inform  budget  requests, 
which  are  made  well  in  advance  of  programmatic  decisions.  Testing  and 
evaluation  could  then  be  cast  in  a  more  constructive  role  of  helping  a 
weapon  system  reach  these  levels  and  would  ease  some  of  the  burden 
currently  placed  on  program  managers  to  rely  on  judgment,  rather  than 
demonstrated  product  maturity,  in  promising  success  at  times  when  major 
funding  commitments  have  to  be  made. 


C.  ANALYSIS  OF  SIMULATION  AND  TESTING  STRATEGY 

Primary  Question:  What  was  the  simulation  and  testing  strategy  for  the  system, 
and  did  that  strategy  adequately  evaluate  the  system  for  its  ultimate  operational  use? 
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1.  Test  Approach  Used? 

Validation  Activities:  Testing  and  Simulation 

In  the  early  stages  of  the  project,  the  contractor  did  a  Failure  Modes  Effects  and 
Criticality  Analysis  (FMECA)  on  the  system.  This  analysis  helped  establish  the  test 
requirements  and  influenced  simulation.  This  in  turn,  drove  the  design  of  test  equipment 
and  field  automatic  test  equipment,  and  they  also  used  the  same  test  equipment  at  the 
Aviation  Intermediate  Maintenance  (A VIM)  facility.  Another  FMECA  result  was  that 
performance  tests  were  run  under  extreme  vibration  conditions.  The  production  planning 
efforts  during  development  focused  on  logistics  in  an  effort  to  identify  the  full  spectrum 
of  support  requirements. 

Testing  and  simulation  (T&S)  were  performed  in  a  variety  of  settings:  T&S  of 
individual  components,  T&S  of  components  in  controlled  settings,  tests  of  components  in 
realistic  settings,  and  a  hardware-in-the-loop  type  systems  integration  simulation 
laboratory. 

The  full  test  strategy  included  tests  of  sub-assemblies  and  of  some  individual 
components,  perfonned  by  the  Prime  and  suppliers  as  appropriate.  They  also  verified 
some  components  with  simulations. 

The  Prime,  their  suppliers,  and  Government  organizations  tested  some  integrated 
components  in  a  controlled  setting.  The  Prime  and  U.S.  Army  labs  perfonned 
simulations  of  some  components  working  together  in  a  controlled  setting.  They  also 
tested  components  working  together  in  a  realistic  setting. 

A  hardware-in-the-loop  type  systems  integration  simulation  laboratory  was  used 
to  check  individual  components  of  the  system,  and  to  check  integrated  components  in  a 
controlled  setting.  TADS/PNVS  used  aircraft  simulator  test  equipment  in  the  aircraft 
manufacturer’s  Systems  Integration  Laboratory. 

Summarizing  the  total  amount  spent  on  the  testing  and  simulation  discussed 
above,  the  contractor  spent  roughly  15%  to  see  if  the  individual  components  of  the 
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system  worked,  5%  to  see  if  integrated  components  worked  in  controlled  setting,  and 
80%  to  see  if  integrated  components  worked  in  a  realistic  setting. 

Validation  work  on  component  and  system  maturity  was  done  in  plenty  of  time  to 
allow  using  that  infonnation  to  help  the  program.  Validation  knowledge  was  used 
consistently  to  improve  components  and  thee  system.  However,  the  project  didn’t  do  all 
they  could  early  on  to  get  rid  of  project  risk.  The  project's  test  philosophy  was  to  “Break 
it  big  early,”  but  sometimes  caution  prevented  using  rigorous  testing. 

Testers  and  project  personnel  on  TADS/PNVS  were  in  a  good  teaming 
relationship.  The  TADS/PNVS  program  used  the  best  validation  methods  available,  and 
they  were  quick  to  recognize  important  lessons  learned  from  this  work.  The  majority  of 
the  project  validation  work  was  of  high  quality. 

2.  GAO  Research: 

“Late  Cycle  Chum:” 

According  to  the  GAO,  within  the  DoD,  it  is  fairly  common  to  have  test  failures 
late  in  the  test  cycle,  which  could  (and  should)  have  been  discovered  earlier  in 
development.  In  industry,  this  is  called  "late-cycle  churn,"  the  scramble  to  fix  a 
significant  problem  discovered  late  in  development.  The  frequent  result  is  spending  a  lot 
more  money  to  fix  problems.  Discovering  and  fixing  problems  earlier  in  development 
costs  far  less  money,  and  also  saves  time.  Civilian  companies  that  develop  products  have 
a  different  testing  and  evaluation  philosophy,  the  “Break  it  Big  Early”  philosophy,  which 
works  to  preclude  "late-cycle  churn." 

Test  Failure  Consequences: 

Rather  than  being  viewed  as  constructive,  some  DoD  organizations  avoid  difficult 
tests  because  test  failures  can  increase  the  probability  of  funding  reductions  or  program 
cancellation.  Testing  is  often  linked  to  funding  and  can  jeopardize  program  support  -  a 
penalty  environment.  However,  delivering  late  and  over  budget  is  not  viewed  as  harshly. 
Of  course,  late  cycle  churn  has  even  greater  cost  and  schedule  impacts,  but  because  of  the 
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length  of  the  normal  developmental  cycle,  it  is  frequently  years  later  and  under  a  new 
program  manager,  developer,  or  tester,  when  faults  are  discovered. 

Even  if  a  program  is  somewhat  protected  by  powerful  ‘champions,’  other  negative 
consequences  are  possible.  These  consequences  or  reactions  from  test  failures,  beside 
funding  reductions  and  program  cancellation,  range  from  getting  called  ‘on  the  carpet’ 
and  reprimanded  unofficially,  to  getting  fired  -  and  possibly  career  ruination.  The 
project  may  be  protected,  but  that  does  not  mean  that  the  program  manager  and  staff  are 
similarly  protected. 

However,  testing  is  perhaps  the  most  effective  tool  for  discovering  design 
problems.  Some  development  critics  want  you  to  fix  everything  before  a  particular  test. 
But  you  have  to  find  problems  before  you  can  fix  them,  and  testing  is  a  great  way  to  find 
problems.  Discovering  problems  in  new  products  is  a  nonnal  part  of  development  and 
known  as  ‘test-fix,  test-fix,  test-fix....’  Eliminating  problems,  which  can  only  be  done 
after  discovery,  improves  a  product.  Testing  corrects  latent  problems  that  could  delay 
product  delivery,  increase  product  cost,  or  be  delivered  to  the  customer.  Field  users 
usually  view  faulty  delivered  products  as  a  blunder. 

There  is  an  apparently  strong  expectation  in  the  military  that  test  failures  will 
cause  strong  negative  reaction  by  somebody  in  their  chain  of  command.  Whether  this 
belief  is  always  justified,  is  not  as  important  as  that  it  is  probably  true  often  enough  to 
cause  the  belief  to  be  as  widespread  as  it  is.  This  expectation  is  a  common  part  of 
military  organizational  culture,  and  as  such,  is  very  difficult  to  change. 

On  the  other  hand,  cancellation  of  a  program  due  to  various  risks  should  be 
minimized.  Viewed  pragmatically,  if  the  risk  due  to  test  failures  is  a  lot  higher  than  due 
to  schedule  or  cost  problems,  then  you  should  not  risk  test  failures.  This  has  been  the 
case  for  quite  a  while. 

Less  Testing: 

There  are  many  people  at  different  levels  who  have  reasons  to  delay  or  diminish 
tests.  In  many  cases,  the  reason  given  is  that  the  test  article  is  not  ready  for  the  test  as 
written,  so  modifying  it  is  very  logical. 
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Development  personnel  are  normally  divided  between  developers  and 
developmental  testers.  This  segregation  helps  keep  the  testers  relatively  independent,  and 
therefore  effective.  But  there  has  to  be  someone  who  is  in  charge  of  them  all,  namely  the 
program  manager.  The  PM  is  graded  on  how  well  the  development  goes.  If  the  PM 
knows  the  system  is  not  ready  for  a  test,  then  it  is  quite  easy  to  modify  some  tests  so  that 
the  system  passes.  You  still  get  some  data,  although  it  is  arguable  how  valuable  the  data 
will  be. 

And  of  course,  if  the  PM’s  next  promotion  may  depend  upon  the  system  passing  a 
test,  then  it  is  hard  to  resist  some  types  of  changes.  Other  changes  include  delaying  tests, 
deferring  difficult  steps  from  tests,  or  lowering  the  test  pass/fail  criteria.  It  is  much  more 
difficult  to  affect  the  results  of  major  tests.  However,  some  programs  go  a  long  time 
between  the  major  tests.  When  the  system  comes  to  a  major  test  and  fails,  it  is  often  a 
surprise  to  those  outside  of  the  program. 

These  test  changes  are  often  rationalized:  if  they  know  a  system  can  not  pass  a 
test,  they  change  the  test  to  something  the  system  can  pass  -  what  is  the  sense  of  running 
a  test  you  know  will  fail?  But,  then  the  passed  test's  results  are  occasionally  substituted 
for  those  of  the  originally  programmed  test.  They  have  lowered  the  test  criteria,  while  (in 
effect)  presenting  these  results  as  proof  that  the  system  is  on  track,  this  does  not  mean 
that  PMs  are  dishonest,  but  that  there  are  routinely  many  testing  modifications,  delays,  or 
de-scopings.  The  end  result  is  that  testing  of  a  system  is  less  rigorous  than  needed  -  not 
all  the  delays  are  due  to  one  person,  or  one  reason  -  and  thus  the  system  is  more  prone  to 
"late-cycle-churn."  Integrity  requires  listing  test  changes  on  test  reports,  to  notify  the 
chain  of  command,  auditors,  and  other  interested  people. 

This  fear  of  performing  tests  to  leam,  engendered  by  the  expectation  of  negative 
reactions  by  higher-level  managers,  causes  the  learning  curve  to  be  lower  in  DoD. 
Testers  ’know’  they  should  not  risk  test  failures  because  of  the  potential  reaction  of  the 
Program  Manager;  PMs  ’know'  they  cannot  risk  high-visibility  test  failures  because  of  the 
Program  Executive  Officer's  reaction;  PEOs  'know'  they  have  to  consider  the  Service 
Acquisition  Executive’s  reaction;  and  the  SAE  'knows'  they  must  consider  the  SecDef 
and  Congressional  reactions  which  could  result  in  a  significant  loss  of  funds,  or  even 
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program  cancellation.  Of  course,  this  is  despite  an  official  policy  to  find  flaws  earlier. 
However,  in  some  cases,  there  appears  to  be  insufficient  energy  behind  this  newer  policy 
to  motivate  certain  echelons  of  the  oversight  chain  to  protect  the  PM’s  honest 
developmental  testing  efforts  from  the  circling  vultures  eager  to  pounce  upon  a  routine 
test  failure  (learning  experience)  as  an  excuse  to  seize  his  funds. 

Break  It  Big  Early  -  Testing  to  Learn: 

There  are  basically  two  types  of  testing:  there  is  testing  to  learn,  and  testing  to 
confirm.  The  first  type  is  done  early  in  development,  but  it  is  frequently  not  used 
sufficiently  by  the  DoD.  In  commercial  development,  the  testing  philosophy  is  to  find 
flaws  and  to  continuously  improve  the  product,  and  make  it  as  robust  as  possible.  DoD 
sees  testing  as  a  "final  exam"  or  an  audit  to  evaluate  a  product.  And  the  culture  of  the 
DoD  often  forces  managers  into  optimistic  plans  that  presume  success,  instead  of 
anticipating  problems.  This  success-oriented  philosophy  is  somewhat  understandable, 
given  the  reluctance  to  risk  test  failures. 

Break  It  Big  Early  (BIBE)  is  a  testing-to-learn  strategy.  It  tells  you  about  your 
system.  If  you  use  this  early  testing  as  a  testing-to-confirm  opportunity,  you  are  going  to 
affect  the  attitude  of  the  author  of  the  test  procedure,  the  tester,  and  the  program  manager. 
You  are  adjusting  their  attitudes,  during  a  preliminary  stage  of  the  design.  It  is  all  very 
well  to  tell  the  PM  to  reduce  risk,  but  this  ‘catch-22’  situation  makes  it  very  difficult. 

Experimental  Test  Pilots  try  to  ’break'  equipment,  that  is  they  tend  to  test  an 
aircraft  on  the  ground  very  rigorously,  especially  new  systems  or  new  features,  trying  to 
do  whatever  they  can  to  'break'  it,  and  then  get  those  failures  fixed.  But  when  they  are 
flying  a  prototype  aircraft,  they  are  more  circumspect,  and  often  will  have  a  list  of 
functions  to  avoid,  and  a  list  of  corrective  procedures  in  case  something  does  malfunction 
or  'break.' 

Commercial  companies  strive  for  short  development  cycles,  usually  less  than  two 
years,  which  improves  their  return  on  investment.  Production  is  seen  as  a  near-term 
event,  so  there  is  more  impetus  to  test  the  product  thoroughly  and  find  all  the  hidden 
flaws. 
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Testing  for  the  Scorecard: 

Testing  in  DoD  becomes  a  scorecard  to  rate  system  developmental  progress.  A 
major  test  is  a  decision  point  on  whether  or  not  to  proceed  with  the  program.  Thus, 
program  managers  have  incentives  to  avoid,  or  at  least  postpone,  risky  testing,  and  to 
limit  open  communications  of  results.  Oversight  organizations  occasionally  try  to  get 
access  to  lower  level  test  data,  although  some  program  managers  may  be  able  to  fend 
them  off.  Program  success  is  frequently  detennined  (especially  by  proponents  in 
Congress  with  constituent  jobs  at  risk)  by  how  well  you  keep  the  money  coming  in,  not  if 
the  product  fulfills  its  requirements.  'Full  funding'  is  not  an  end  in  itself;  the  system  must 
fulfill  its  requirements,  as  verified  by  testing. 

In  both  DoD  and  commercial  companies,  testing  is  conducted  by  separate 
organizations.  The  separation  makes  testers  independent,  and  hopefully  more  candid. 
However,  the  negative  side  of  the  separation,  is  testers  have  less  say  in  the  design; 
testability  is  not  designed  in.  Testers  look  at  requirements  from  the  point  of  view  of  how 
they  will  be  tested,  which  is  a  useful  perspective  and  a  design  tool  that  DoD  often 
overlooks.  Also,  in  the  DoD  there  is  often  an  adversarial  relationship  between  developers 
and  testers,  particularly  the  Operational  Testers. 

GAO  recommends  increasing  levels  of  maturity  earlier  in  the  development  cycle, 
and  proposes  various  methods  to  carry  this  out.  However,  the  negative  incentives 
surrounding  rigorous  early  tests  still  remain  in  DoD.  If  acquisition  managers  are  going 
pursue  a  higher  level  of  maturity,  they  will  have  to  decrease  the  risk  of  test  failures 
resulting  in  funding  losses.  This  will  not  be  easy  to  do.  The  GAO  study  quotes  a 
September  1999  report  from  the  Defense  Science  Board:  “The  response  to  perceived  test 
failures  is  often  inappropriate  and  counterproductive.”  The  GAO  continues: 

Instead  of  using  testing,  especially  in  the  early  stages,  as  a  vital  learning 
mechanism  and  an  opportunity  to  expand  product  knowledge,  testing  is 
often  used  as  a  basis  for  withholding  funding,  costly  rescheduling,  or 
threats  of  cancellation. 
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D.  CONCLUSIONS  ON  SIMULATION  AND  TESTING  STRATEGY 

Primary  Question:  What  was  the  simulation  and  testing  strategy  for  the  system, 
and  did  that  strategy  adequately  evaluate  the  system  for  its  intended  ultimate  operational 
use? 


1.  Test  Approach  Used? 

Validation  Activities:  Testing  and  Simulation 

The  contractor  performed  a  Failure  Modes  Effects  and  Criticality  Analysis 
(FMECA),  and  it  strongly  impacted  design,  testing,  test  equipment,  and  simulation.  The 
testing  and  simulation  (T&S)  had  considerable  variety.  Various  organizations  did  tests  of 
components,  sub-assemblies,  and  the  end  system;  some  components  were  verified  by 
simulations.  They  tested  components  and  sub-assemblies  in  controlled  settings 
(hardware-in-the-loop),  and  sub-assemblies  in  a  realistic  setting  (aircraft  simulator  test 
equipment).  The  vast  majority  of  the  test  money  was  spent  on  the  last  category,  with  a 
modest  amount  on  testing  in  controlled  settings. 

Fairly  high  quality  validation  work  helped  the  program  in  system  maturity  and 
improving  components,  but  some  early  risks  were  not  handled  as  well  as  they  should 
have  been.  This  is  possibly  an  instance  of  avoiding  testing  that  might  fail  and 
consequently  threaten  the  program’s  continued  funding.  However,  testers  and  project 
personnel  usually  worked  well  together. 

2.  GAO  Research: 

The  Best  Practices  GAO  Report  shows  that  DoD  programs  routinely  suffer  from 
“Late  Cycle  Churn,”  a  condition  describing  operational  testing  failures  that  discovered 
problems  that  should  have  been  revealed  in  earlier  developmental  testing.  Commercial 
developers  use  the  “Break  it  Big  Early”  philosophy.  It  allows  them  to  uncover  system 
problems  earlier,  when  they  are  less  expensive  to  fix. 

But  DoD  is  reluctant  to  run  a  test  that  they  might  fail,  even  if  it  might  highlight 
potential  flaws.  Test  failure  could  cause  the  program  to  loose  some  of  its  funding  or  be 
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cancelled  -  a  harsh  penalty.  It  also  reflects  poorly  on  the  Program  Managers,  to  the 
extent  that  they  may  never  be  promoted  again.  Late  deliveries  and  cost  overruns  are 
more  common  and  not  so  harshly  punished.  If  DoD  hopes  to  modify  this  balance,  they 
need  to  also  modify  the  probabilities  (and  severity)  of  rewards  and  punishments. 

DoD  does  less  testing  early  in  development,  which  impacts  the  readiness  of  the 
system  in  late-cycle  operational  testing.  The  program  manager,  developers,  and  testers 
should  not  be  judged  solely  on  developmental  success,  but  also  on  running  a  thorough 
developmental  test  program.  Otherwise,  there  is  a  great  incentive  to  run  cursory  tests. 

Testing  is  a  value-added  activity,  a  necessity  for  successful  development.  The 
more  rigorous  the  testing,  the  more  you  find  out  about  the  test  article.  You  have  to  find 
flaws  in  order  to  fix  them.  But  testing-to-learn  should  not  be  used  for  a  'final  exam’  or 
'Scorecard'  opportunity. 

Simply  ordering  program  managers  to  test  more  thoroughly  will  probably  not  be 
effective  for  DoD  in  the  long  run.  Any  programs  that  increase  testing  will  undoubtedly 
uncover  more  problems;  however,  if  those  programs  face  the  threat  of  being  cancelled, 
then  future  program  managers  will  get  the  message  that  more  thorough  testing  will  not 
save  their  programs.  DoD  must  change  the  definition  of  what  constitutes  success  in 
weapon  systems  and  increase  constructive  testing  and  evaluation. 

The  testing  philosophy  of  the  TADS/PNVS  program  was  not  very  different  than 
those  of  other  defense  contractor  companies  at  the  time.  The  TADS/PNVS  was  a  very 
important  part  of  the  AAH  program,  but  very  risky.  It  is  likely  that  upper  DoD  managers 
would  have  balked  at  the  contract  if  it  had  presented  a  realistic  cost  estimate  (based  upon 
“20-20  hindsight.”)  But  the  lower  cost  caused  some  risky  behavior  -  you  cannot  allocate 
money  to  risk  reduction  testing  if  funds  are  not  available. 
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E.  RECOMMENDATIONS  FOR  FURTHER  STUDY 

1.  Test  Strategies: 

Study  how  the  merging  of  Developmental  Testing  with  Operational  Testing  has 
affected  the  openness  of  the  program  management  offices  and  development  contractors  to 
the  “break  it  big  early”  philosophy. 

2.  NASA  Development  Test  Strategies: 

Study  how  the  NASA  compares  with  the  DoD  and  commercial  developers  in  their 
testing  strategy.  Do  these  developers  compare  more  to  the  commercial  model,  or  the 
military  model.  What  are  the  effects  of  large  Government  bureaucracy  on  test  strategies? 
Also,  what  are  the  political  factors  affecting  testing  strategy?  Does  NASA  have  more 
control  of  its  funding,  or  is  it  just  as  subject  to  cancellation  if  a  system  has  poor  test 
results? 


3.  Independent  Research  and  Development  (IR&D): 

Some  programs  have  taken  the  route  of  off-loading  more  and  more  of  their  high- 
risk  work  to  separate  IR&D  projects.  This  reduces  overall  program  risk,  and  test  'failures' 
can  be  disassociated  from  the  program.  These  strategies  may  help  programs  lower  their 
risk  of  losing  funding,  while  reducing  developmental  risk.  Study  how  programs  do  this, 
what  percentage  of  work  is  off-loaded,  and  their  various  strategies  for  integrating  results 
back  into  the  system. 

4.  Commercial  Versus  DoD  Developmental  Programs: 

Study  commercial  and  DoD  developmental  programs  and  compare  them  on  the 
basis  of  program  size,  program  development  time  period  and  program  complexity.  This 
will  supplement  the  overall  comparison  of  all  programs.  Are  the  DoD  programs  of 
similar  size  /  complexity  /  development  time  as  efficient  as  commercial  programs? 
Compare  how  well  each  has  adopted  lean  engineering  and  lean  manufacturing 
techniques. 


76 


LIST  OF  ACRONYMS  AND  ABBREVIATIONS 


AAH  AH-64  Apache  Attack  Helicopter 

ACAT  Acquisition  Category 

AMC  U.S.  Army  Material  Command 

AMCOM  U.S.  Army  Aviation  and  Missile  Command 
ANSI  American  National  Standards  Institute 

ASH  Advanced  Scout  Helicopter  -  cancelled  in  the  1970s 

ATCOM  U.S.  Army  Aviation  and  Troup  Command  (ATCOM  and  MICOM 
are  now  merged  into  AMCOM) 

D  Development  (Question  on  the  Survey) 

DoD  Department  of  Defense 

DOS  Disk  Operating  System 

DPM  Deputy  Program  Manager 

DSMC  Defense  System  Management  College 

FLIR  Forward-Looking  Infra-Red 

FT  A  FLIR  T  arget  Ac  quisition 

G&C  Guidance  and  Controls  Laboratory 

GAO  General  Accounting  Office 

IPT  Integrated  Product  Team 

IBM  International  Business  Machines 

IR&D  Independent  Research  and  Development 

LOS  Line-of-Sight 

LOSS  Line-of-Sight  Stabilization 

LSBS  Laser  to  sensor  bore-sight 

MICOM  U.S.  Army  Missile  Command(ATCOM  and  MICOM  are  now 
merged  into  AMCOM) 

MIT  Massachusetts  Institute  of  Technology 

NPS  Naval  Postgraduate  School 

NVL  Night  Vision  Laboratory 

OS  Operating  System 

ODS  Operation  Desert  Storm 

PM  Program  Manager 

PMO  Program  Manager’s  Office 

PNVS  Pilot  Night  Vision  System 

S&T  Science  and  Technology  Group  within  system  Prime  Contractor 

responsible  for  doing  IR&D  and  developing  new  technology 
and  concepts. 

SP  Start  of  Program  (Question  on  the  Survey) 

TADS  Target  Acquisition  Designation  System 

Technology  A  Line-of-Sight  Stabilization 
Technology  B  FLIR  target  acquisition 
Technology  C  Laser  to  sensor  bore-sight 

TP  Transition  to  Production  (Question  on  the  Survey) 
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TRADOC 

TRL 

UAH 


U.S.  Army  Training  &  Doctrine  Command  (and/or  other 
appropriate  user  representatives) 

Technology  Readiness  Level 
University  of  Alabama  in  Huntsville 
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APPENDIX  A:  COMBINED  QUESTIONNAIRE  RESPONSE: 

The  combined  survey  was  created  by  combining  the  answers  from  Government 
and  contractor  responses. 
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Combined  Survey 


©  University  of  Alabama,  Huntsville,  12/10/02  0 

Desert  Storm  Case  Study  Checklist:  Lessons  for  Technology  Management 

The  U.S.  Army  Materiel  Command  is  supporting  a  hindsight  study  of  how  technologies  were 
developed,  integrated  into  systems,  and  produced  in  the  years  leading  up  to  Desert  Storm,  the  last 
large-scale  deployment  of  U.S.  military  force.  It  is  believed  that  in  the  years  leading  up  to  that 
conflict,  there  were  both  successful  and  unsuccessful  applications  of  technology  to  military  systems 
that  contain  lessons  for  future  defense  technology  development.  The  study  can  be  done  now  because 
the  intervening  years  allow  more  objectivity,  and  allow  open  examination  of  what  were  once 
classified  projects.  The  study  must  be  done  now  because  many  of  the  men  and  women  responsible  for 
the  development  and  eventual  fielding  of  those  systems  in  Gulf  region  are  retiring,  taking  with  them 
important  knowledge  that  we  believe  should  be  captured  and  codified  into  practical  lessons  for  the 
future. 

Our  method  began  with  a  list  of  military  systems  including  both  successes  and  failures  judged  to  be 
broadly  representative  of  the  systems  that  were  under  development  in  the  years  prior  to  Desert  Storm. 
Then  experienced  students  (such  as  those  found  at  senior  military  schools  and  mid-career  management 
programs)  are  being  asked  to  create  a  single  case  study  for  a  project  on  that  list.  Each  case  will  include 
both  (1)  a  narrative  case  history  to  capture  the  richness  of  the  case  and  identify  any  factors  that 
determined  a  project’s  success  or  failure,  and  (2)  answers  to  structured  questions  that  ask  about 
organization,  technology  and  process  issues  in  a  consistent  way  across  all  cases. 

Participants  in  the  selected  projects  are  being  asked  to  complete  this  survey  form  as  background 
information  for  the  students  to  use  in  their  projects,  and  we  hope  you  can  cooperate  with  our  research. 

This  is  not  a  traditional  questionnaire.  If  you  do  not  remember  the  details  we  are  asking  about,  or  if 
you  feel  that  the  answer  would  be  misleading  or  somehow  inappropriate  for  the  project  we  are  asking 
you  about,  feel  free  to  leave  the  answer  blank.  You  may  rewrite  the  question  so  it  fits  better.  If  you 
have  comments  to  add,  or  want  to  suggest  a  better  answer  than  what  is  provided,  feel  free  to  do  so. 

While  the  students  conducting  this  research  may  be  cleared  to  discuss  classified  material,  it  should  be 
stressed  that  the  narratives  and  the  answers  to  structured  questions  should  never  include  any  classified 
information.  The  results  will  be  used  in  unclassified  reports. 

You  may  request  a  copy  of  any  report  of  the  findings  by  providing  your  business  card,  or  providing  a 
separate  sheet  of  paper  with  your  name  and  address  information,  including  your  e-mail  address.  If 
you  have  any  questions,  contact: 

Richard  G.  Rhoades  William  A.  Lucas 

Research  Institute  Sloan  School  of  Management 

The  University  of  Alabama  in  Huntsville  Massachusetts  Institute  of  Technology 

Rhoadesrtaiemail.uah.edu  (256)  S24-6343  walucas@mit.edu  (617)  253-053S 

TO  BEGIN:  The  first  set  of  questions  defines  three  dates,  keyed  to  technology  readiness  levels 
(see  page  8),  and  then  asks  about  the  roles  played  by  different  organizations  at  three  stages  of 
your  proj  ect  leading  up  to  those  dates .  The  organizations  of  interest  are : 

Prime’s  S&T  org.:  Group  within  system  prime  contractor  responsible  for  doing  IR&D  and 

developing  new  technology  and  concepts. 

Other  prime  org. :  Any  prime  contractor  organization  other  than  the  S&T  organization. 

Supplier  S&T:  Same  definition  as  for  prime’s  S&T  organization,  but  located  at  a  supplier. 

Other  supplier  org. :  Any  supplier  organization  other  than  the  S&T  organization. 

Army  Lab/Center:  One  or  more  of  the  Army  laboratories  or  research,  development  and  engineering 

Centers. 

Other  DoD/S&T  org.:  An  equivalent  of  an  Army  Lab/Center  found  elsewhere  in  DoD. 
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SP.  What  was  the  approximate  starting  date  of  systems  planning  and  pre-development  work?  This  date  is  when 
planning  work  began  on  the  integrated  system.  The  systems  concept  and  applications  had  been  formulated, 
but  applications  were  still  speculative.  There  was  no  proof  or  detailed  analysis  to  support  the  approach. 
SYSTEMS  PLANNING  START  DATE  (SP):  _ /1976  (mo/yr)  [TRL2  at  system  level] 

In  what  organization  was  the  primary  work  leading  up  to  this  point  accomplished?  There  were  really  three 
important  organizations  that  contributed.  The  ASH  PM  (Advanced  Scout  Helicopter)  prior  to  being 
cancelled  and  the  AAH  PM  were  the  driving  force  for  establishing  and  planning  the  program.  The 
technology  work  was  being  led  by  the  MICOM  G&C  lab  and  The  Night  Vision  Lab  (NVL).  Contractor 
S&T  organizations  were  doing  their  own  work  in  response  to  the  anticipated  requirement  for  the  ASH  and 
AAH  programs. 


Including  that  organization,  what  organizations  had  been  involved  up  to  this  point?  (Check  the  role  of  each) 


Prime’s  S&T  org. 

_ Lead/co-lead 

X  Active  support 

_ Involved 

_ Kept  informed 

_ Not  involved 

_ DK 

Other  prime  org. 

_ Lead/co-lead 

_ Active  support 

_ Involved 

_ Kept  informed 

_ Not  involved 

_ DK 

Supplier  S&T  org. 

_ Lead/co-lead 

_ Active  support 

_ Involved 

_ Kept  informed 

_ Not  involved 

_ DK 

Other  supplier  org. 

_ Lead/co-lead 

_ Active  support 

_ Involved 

_ Kept  infonned 

_ Not  involved 

_ DK 

Army  Lab/Center 

_ Lead/co-lead 

X  Active  support 

_ Involved 

_ Kept  informed 

_ Not  involved 

_ DK 

Other  DoD/S&T  org. 

_ Lead/co-lead 

_ Active  support 

_ Involved 

_  X  Kept  informed 

_ Not  involved 

_ DK 

What  was  the  nature  of  the  Army  Lab/Center’s  involvement?  (Simulation?  Concept  formulation?  Integration? 
Requirements  development?)  The  MICOM  G&C  lab  was  the  developer  of  the  Hellfire  missile  for  which  the 
TADS  acquires  and  designates  targets.  Major  systems  requirements  such  as  Total  Pointing  Error  (TPE)  for 
the  laser  designator  were  defined  by  the  G&C  lab  based  on  testing  and  simulations.  They  also  did  early 
work  on  the  laser  hardware  that  does  the  designation.  NVL  was  the  developer  of  FLIR  technology  which 
was  used  in  the  TADS  night  side  and  the  PNVS.  They  were  responsible  for  the  development  and  eventually 
production  of  the  FLIR  common  modules  which  are  used  in  the  TADS  Night  Side.  Signifiicant  support  w  as 
given  by  these  labs  and  Frankfort  Arsenal  (fire  control,  optics)  in  formulating  requirements,  evaluating 
proposals  and  monitoring  development  progress. 


D.  Date  when  Development  started.  Typically  at  this  date,  funding  started  for  system  advanced  or  engineering 
development,  a  gov’t  project  office  was  formed  and  prime  contractor(s)  selected. 

DEVELOPMENT  START  DATE:  (D):  _ 11911  (mo/yr) 

What  was  the  Technology  Readiness  Level  (refer  to  page  8)  for  the  SYSTEM  on  this  date?_  Level  3 

What  was  the  Production  Readiness  Level  (refer  to  page  8)  for  the  SYSTEM  on  this  date?  Level  1 

In  what  organization  was  the  primary  work  in  the  period  from  SP  to  D  accomplished?  Supplier 

Including  that  organization,  what  organizations  had  been  involved  in  the  period  SP  to  D?  (Check  the  role  of  each.) 

Prime’s  S&T  org  _ Lead/co-lead  _X  Active  support  _ Involved _ Kept  informed _ Not  involved  _ DK 

Other  prime  org  _ Lead/co-lead  _ Active  support  _X  Involved _ Kept  informed _ Not  involved  _ DK 

Supplier  S&T  _X  Lead/co-lead  _ Active  support  _ Involved  _ Kept  informed _ Not  involved  _ DK 

Other  supplier  org  _ Lead/co-lead  _ Active  support  Involved  _ Keptinfoimed  _ Not  involved _ DK 

Army  Lab/Center  _ Lead/co-lead  X  Active  support  Involved _ Kept  informed _ Not  involved  _ DK 

Other  DoD/S&T  org.  _ Lead/co-lead  _ Active  support  _X  Involved  _ Kept  informed _ Not  involved  _ DK 


What  was  the  nature  of  the  Army  Lab/Center’s  involvement?  (Engineering  support?  Simulation  or  testing? 
Integration?  Requirements  interpretation?)  G&C  Lab,  NVL,  and  Frankfort  Arsenal  provided  engineering 
support,  simulation,  and  requirements  interpretation. 


TP.  Date  of  achieving  “Transition  to  Production”  when  producible  system  prototype  has  been  demonstrated  in  an 
operational  environment.  Prototype  is  near  or  at  planned  operational  system,  produced  on  small  scale. 

TRANSITION  TO  PRODUCTION  (TP)  DATE:  /1980  (mo/yr)  (TRL7  at  system  level) 

What  was  the  Production  Readiness  Level  for  the  SYSTEM  on  this  date?  Level  2 

In  what  organization  was  the  primary  work  in  the  period  from  D  to  TP  accomplished?  Martin  Marietta,  Orlando, 
the  prime  contractor.  This  work  was  done  under  a  project  management  (PM)  organization. 
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Including  that  organization,  what  organizations  had  been  involved  in  the  period  D  to  TP?  (Check  the  role  of  each.) 

Prime’s  S&T  org  _ Lead/co-lead  _X  Active  support  _ Involved  _ Kept  informed _ Not  involved  _ DK 

Other  prime  org  _ Lead/co-lead  _ Active  support  _X  Involved _ Kept  informed  _ Not  involved  _ DK 

Supplier  S&T  _X_  Lead/co-lead  _ Active  support  _ Involved  _ Kept  informed _ Not  involved  _ DK 

Other  supplier  org  _ Lead/co-lead  _  X  Active  support  _ Involved  _ Kept  informed _ Not  involved  _ DK 

Army  Lab/Center  _ Lead/co-lead  _X  Active  support  _ Involved _ Kept  informed _ Not  involved  _ DK 

Other  DoD/S&T  org.  _ Lead/co-lead  _ Active  support  _ Involved  _X  Kept  informed _ Not  involved  _ DK 

What  was  the  nature  of  the  Army  Lab/Center’s  involvement?  (Engineering  support?  Simulation  or  testing? 
Integration?  Requirements  interpretation?)  MICOM  G&C,  NVL,  and  Frankfort  Arsenal  continued  to 
protide  engineering  support,  simulation,  test  witnessing  and  requirements  interpretation. 

Please  note:  Here  we  shift  away  from  the  system  as  a  whole,  and  move  to  its  component  technologies. 

Tl.  Now  identify  one  or  more  (up  to  3)  technologies  that  were  incorporated  into  the  system  you  are  studying. 

These  technologies  should  be  among  those  central  to  the  success  of  the  system. 

Technology  A  Line  of  Sight  Stabilization 
T echn ologv  B  FLIR  target  acquisition 
Technology  C  Laser  to  sensor  bore  sight 


T2.  How  new  was  each  technology  to  the  prime  contractor?  For  each  technology  A,  B,  and  C,  was  the 
technology: 

(Answer  for  date  you  provided  for  Development  start ,  D.)  Technology  A  Technology  B  Technology  C 


New  and  unproven  for  the  prime  contractor?  1.  _X 

Technology  had  been  used  by  prime  contractor  but  it  was  new  to 

to  this  kind  of  application?  2. _ 

Technology  had  been  used  by  prime  contractor  in  similar  applications 

and  was  weii  understood?  3. _ 

Don’t  know,  can’t  remember,  or  would  have  to  guess  9. _ 


1.  _  l._X. 

2.  _X_  2. _ 

3.  _  3. _ 

9.  9. 


T3.  Production  Impact.  What  was  the  impact  of  the  technology  on  then  existing  production  processes? 


(Answer  for  date  you  provided  for  Development  start ,  D.) 

Technology  A 

Technology  B 

Technology  C 

Technology  forced  deep  and  serious  production  process  change? 

l.X_ 

1.  x  _ 

i._ 

Technology  caused  significant  production  process  change? 

2. _ 

2. _ 

2.  X  _ 

Technology  did  not  require  much  production  process  change 

3. _ 

3. _ 

3. _ 

Don’t  know,  can’t  rem ember,  or  would  have  to  guess 

9.  _ 

9. _ 

9.  _ 

T4.  Looking  back  at  the  Development  start  date,  at  that  time  how  important  were  these  technologies  to  the 
prime? 


Check  (V)  the  best  answer  for  each  technology.  T echnologv  A 

This  system  was  the  Prime’s  only  planned  application  ofthe  technology.  1. _ 

Prime  was  planning  or  had  started  follow-on  uses  of  the  technology.  2.  _  X  __ 
Technology  was  being  used  in  other  applications  and  it  was 

expected  to  be  significant  area  of  competence  for  the  Prime.  3. _ 

Don’t  know,  can’t  remember,  or  would  have  to  guess.  9. _ 


Technology^  T  eehnology  C 
1. _  1. _ 

2.  X _  2.  _X 

3.  _  3. _ 

9.  9. 


Now  look  at  the  SP,  D,  and  TP  dates  you  provided  above  for  the  System.  Using  the  Technology  Readiness  iTRL) 
Scale  on  page  8,  find  the  number  that  represents  the  readiness  of  the  separate  technologies  the  team  was  working 
with  at  each  point  in  time.  Please  answer  here  for  the  state  of  development  of  each  component  technology.  (NOT 
for 

tile  over-all  system  which  was  the  focus  in  the  questions  above.)  Technoloirv  A  Technology  B  Technology  c 
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T5.  When  System  planning  and  pre-development  began,  technology  TRL  was:  # _ 4 _  # _ 4 _  # _ 3 _ 

T6.  When  System  went  into  Development,  technology  TRL  was:  # _ 5 _  # _ 5 _  # _ 4 _ 

T7.  When  System  reached  Transition  to  Production,  technology  TRL  was:  # _ 9 _  # _ 9 _  # _ 8 _ 


When  SP  started  stabilization  technology  was  not  new  and  h  ad  had  m  any  applications  but  none  to  this  difficult  an  application 
in  a  helicopter  flight  envioronment.  Likewise,  due  to  the  work  on  FLIR  technology  by  NVL  there  was  a  significant 
technology  base  to  draw  on;  however,  meeting  target  detection  and  recognition  requirements  was  a  very  difficult  goal  and 
integration  of  a  FLIR  meeting  these  requirements  into  the  stabilized  turret  was  areal  challenge.  The  boresight  problem  was 
recognized  as  critical  from  the  very  beginning  but  achieving  a  boresight  which  met  accuracy  requirements  and  remained 
stable  over  environmental  extremes  proved  very  difficult  and  tenuous.  Since  boresight  stability  is  impacted  by  many  factors 
in  all  sensors,  boresighting  components  and  the  stabilized  turret,  it  was  not  possible  to  address  boresight  shortcomings  until 
the  entire  TADS  system  was  designed,  built,  and  tested  for  other  areas  of  performance. 

T8.  For  each  of  the  technologies  A,  B  &  C,  did  an  Army  Laboratory  or  Center  make  a  significant 
contribution  to  achieving  any  of  the  above  levels  of  technology  readiness? 

Technology  A  Technology  B  Technology  C 


T8.  Yes,  it  contributed  to  Readiness  at  start  of  Planning/Pre-development. 

8a  _ 

8b. 

_X_ 

8c. _ 

T9.  Y es,  it  contributed  to  Readiness  for  Development. 

9a.  _X_ 

9b. 

_X_ 

9c.  _X  _ 

T10.  Yes,  it  contributed  to  Readiness  for  Transition  to  Production. 

o 

p 

1 

X 

1 

10b. 

_x  _ 

i 

X 

i 

u 

o 

Tn.  No,  an  Army  lab  or  center  did  not  make  a  significant  contribution. 

No  a _ 

No-b 

No-c _ 

Tdk.  Don’t  know,  can’t  say,  don’t  remember. 

DKa. _ 

DKb 

DKc _ 
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Project  History,  Staffing  and  Location 

HI.  At  some  point,  was  the  project  either:  _ 1.  Slowed  down?  _ 2.  Stopped  and  restarted?  X_  3.  Neither 

TADS/PNVS  program  was  originally  part  of  the  ASH  (Advanced  Scout  Helicopter)  program,  which  was  cancelled.  This 
happened  prior  to  1977,  the  start  of  SP  phase. 

AAH  (Apache  AH  64  PMO)  was  already  involved  when  ASH  left  the  program,  as  was  MICOM.  AAH  and 
MI  COM  support  of  TADS/PNVS  continued  on,  when  ASH  left  the  program. 

H2.  Was  the  project  set  up  as  a  cross-functional  integrated  product  team  (IPT),  a  project  team  drawn  from  different 

parts  of  the  contractor’s  organization  with  most  of  the  skills  needed  for  the  development?  _X  Yes  _ No 

If  YES,  was  it:  _  X  1.  Set  up  by  management,  with  different  functions  &  departments  tasked  to  provide  team  members. 

_  2.  Set  up  informally,  with  team  expected  to  ask  departments  for  help  as  needs  emerged. 

FROM  INTERVIEW:  This  concept  of  an  integrated  product  team  (IPT),  really  came  more  into  vogue  about  or  after  the  time 
we  first  went  into  production.  At  that  time  there  was  a  big  emphasis  to  bring  in  production  people,  logistics,  and,  so  forth  in 
the  very  early  stages  of  the  design  program.  In  the  early  stages  of  the  TADS  PNVS  program,  we  did  have  those  people 
involved  and  that  is  when  I  would  say  we  did  it.  We  did  not  call  it  IPT  and  they  didn’t  organize  it  that  much,  but  there  were 
reliability,  logistics,  maintenance,  and  production  requirements.  In  the  beginning  of  the  program  in  1977  when  they  did  this 
initial  primary  design  with  7  different  contractors  and  then  the  fly  off,  there  was  much  heavier  emphasis,  of  course,  on  the 
performance  aspects  of  TADS/PNVS  because  it  is  something  that  no  one  had  ever  done  before,  so  the  rest  of  it  doesn't  matter 
if  you  cannot  do  the  performance  part. 

H3.  Key  Skills.  This  question  asks  about  “key  skills”  essential  to  the  success  of  the  project,  defined  as  skills 
“that  if  they  were  not  available  at  all,  would  have  stopped  team  progress  at  the  point  when  they  were  needed.” 

Were  there  any  key  skills  not  adequately  represented  on  the  team?  _XNo.  _ Yes,  one.  _ Yes,  more  than  one. 

IF  YES:  H35.  What  were  the  missing  key  skills?  Please  check  (/ )  any  and  all  that  apply. 

_ 1.  Internal  technical  professionals  _ 4.  Technical/development  people  from  Suppliers 

_ 2.  Producibility  professionals  (DFM,  other)  _ 5.  Producibility  professionals  from  Suppliers 

_ 3.  Financial/contracts  professionals  _ 6.  Other.  Please  specify _ 

The  design  chief  could  draft  people  from  other  groups.  It  was  as  good  as  “DX  brick  bat”  priority,  which  same  individual  had 
later  on,  at  least  within  the  company  in  Orlando.  However,  they  didn’t  have  the  microwave  electronics  hybrids  design  group, 
nor  die  printed  circuit  layout  design  people  on  the  project.  Those  were  both  functional  groups,  and  the  TADS/PNVS  group 
didn’t  have  enough  work  to  justify  keeping  them  on  their  team.  But  they  had  as  good  of  a  priority  with  these  groups  as  any 
other  project  in  the  company  in  Orlando. 

H4.  During  the  Development  stage  of  the  project,  how  many  people  on  the  team  were  collocated  very  close 
together?  (On  the  same  floor  of  abuilding  within  a  one  minute  walk.) 

_ 1.  AH  _  X  2.  Most  (2/3rds  or  more)  _ 3.  Some  (over  athird)  _ 4.  Few  _ 5.  None  _ DK/can’t  say 

H4a  Including  the  above,  how  many  people  on  the  team  were  collocated  in  the  same  building? 

_ 1.  All  _ X2.  Most  (2/3rds  or  more)  _ 3.  Some  (over  a  third)  _ 4.  Few  _ 5.  None  _ DK/can’tsay 

Most  were  in  the  same  building,  about  90%,  and  the  rest  were  in  another  building  in  the  same  city. 

H5.  How  many  people  on  the  team  involved  in  the  Development  stage  had  worked  before  with  others  on  the  project? 

_ 1.  All  X  _2.  Most  (2/3rds  or  more)  _ 3.  Some  (over  athird)  _ 4.  Few  _ 5.  None  _ DK/can’tsay 

H6.  Whose  facilities  were  going  to  be  the  primary  production  site  for  the  application  of  the  new  technologies? 

X  _1.  Prime  contractor’s  facilities  _ 2.  Both  prime  and  supplier  facilities  _ 3.  Supplier  facilities 

Validation  Activities:  Testing  and  Simulation 

VI.  Was  a  failure  modes  and  effects  analysis  done  on  the  system?  1. _ X _ Yes  2. _ No  3. _ Don’tknow 

Via  If  yes,  was  it  used  to  help  establish  the  test  plan?  1. X Yes  2. _ No  3.  _ Don’tknow 

This  analysis  drove  the  test  requirements  for  production  test  equipment  and  for  fielded  automatic  test  equipment.  The 
FMECA  looks  for  things  likely  to  break,  and  its  results  influenced  qualification  tests  and  simulation.  Performance  tests  were 
run  under  extreme  vibration  conditions.  Production  planning  was  done  during  development,  as  a  logistics  effort  to  identify 
support  requirements.  They  used  the  same  test  equipment  in  production  as  at  Aviation  Intermediate  Maintenance  (A VIM) 
facility. 
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For  individual  components: 

V2.  Was  there  testing  to  see  if  the  individual  components  of  the  system  worked?  What  organization(s)  did  this  testing? 

_X _ Prime  _ X_Suppliers  _ Army  center/lab _ Othergovt  org.  _ Not  done  on  project _ Don’tknow 

V3.  Were  there  simulations  run  to  see  if  the  individual  components  of  the  system  worked?  What  organization(s) 
did  these  simulations? 

_X _ Prime  _X _ Suppliers  _ Army  center/lab _ Othergovtorg.  _ Not  done  on  project _ Don’tknow 

For  integrated  components  in  controlled  setting: 

V4.  Were  the  components  tested  working  together  in  a  controlled  setting?  What  organization(s)  did  this  testing? 

X _ Prime  _X _ Suppliers  _ Army  center/lab  _X _ Other  govt  org.  _ Not  done  on  project _ Don’tknow 

V5.  Were  there  simulations  of  the  components  working  together  in  a  controlled  setting?  What  organization(s)  did  this? 

_ Prime  _ Suppliers  _  X  Army  center/lab  _ Othergovtorg.  _ Not  done  on  project _ Don’tknow 

For  integrated  components  in  a  realistic  setting: 

V6.  Was  there  testing  of  the  components  working  together  in  a  realistic  setting?  What  organization(s)  did  this  testing? 

_ Prime  _ Suppliers  _  X  Army  center/lab  _ Othergovtorg.  _ Not  done  on  project _ Don’tknow 

V7.  Was  ahardware-in-the-loop  type  systems  integration  simulation  laboratory  used? 

V7a  To  see  if  the  individual  components  of  the  system  worked:  _ X l.Yes  _ 2.No  _ 9.  Don’tknow 

V7h.  To  see  if  integrated  components  worked  in  controlled  setting:  X l.Yes  _ 2.  No  _ 9.  Don’tknow 

TADS/PNVS  components  were  tested  in  the  completed  system  using  "aircraft  simulator”  test  equipment.  The  TADS/PNVS 
itself  was  testing  in  the  aircraft  manufacturer’s  Systems  Integration  Laboratory. 


V8.  Recalling  the  total  effort  (100%)  spent  on  testing  and  simulations,  please  allocate  the  percent  of  that  total  that  were: 

15 _ %  spent  to  see  if  the  individual  components  of  the  system  w'orked 

_5 _ %  spent  to  see  if  integrated  components  worked  in  controlled  setting 

80 _ %  spent  to  see  if  integrated  components  worked  in  a  realistic  setting 

_ %  spent  on  any  other  validation  purpose. 

100  % 


Please  evaluate  the  following  statements  Strongly  Disagree  Neither  agree  Agree  Strongly  Don’t 

about  the  use  Of  testing  and  simulations  on  the  project.  disagree  somewhat  nor  disagree  somewhat  agree  know 


V9.  Knowledge  from  validation  work  was  used  consistently 

to  improve  components  and  system.  1 

V10.  Project  test  philosophy  was  to  “Break  it  big  early.”  1 

Vll.  Component  and  system  maturity  were  validated  at  the  right 

times  in  the  program.  1 

VI 2.  The  project  and  the  testing  community  had  an  adversarial 

relationship.  X 

V13.  Most  project  validation  events  produced  quality  results.  1 

V14.  The  project  didn’t  recognize  important  lessons  that  validation 

work  uncovered.  X 

V15.  Sometimes  the  project  settled  for  less  than  the  best 

validation  method.  X 


2 

2 

2 

2 

2 

2 

2 


3 

3 

3 

3 

3 

3 

3 


4X9 
X  5  9 

4X9 

4  5  9 

X  5  9 

4  5  9 

4  5  9 


Team  Participants  &  Communications  during  Development 

Here  are  some  statements  about  the  people  on  the  project  during  the  System  Development  stage.  Please  circle  anumberto 
indicate  your  level  of  agreement  or  disagreement  that  each  statement  is  a  description  of  team  processes  on  this  project. 

Strongly  Disagree  Neither  agree  Agree  Strongly  Don’t 
disagree  somewhat  nor  disagree  Somewhat  Agree  know 

Dl.  The  team  leader  was  good  at  resolving  technical  disagreements.  1  2  3  4  5  X9 

D2.  The  team  leader  was  good  at  getting  necessary  resources.  1  2  3  4X  5  9 
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D3.  There  was  a  lot  of  turn-over  in  team  membership.  1 

2X 

3 

4 

6 

5  9 

D4.  The  team  leader  had  both  design  &  production  experience.  1 

2 

X 

4 

5  9 

Developer  team  leader  had  excellent  design  experience;  but  production  experience  was  associated  with  smaller  systems. 

D5.  The  team  leader  had  very  high  technical  competence.  1  2  3  4  5  X9 

D6.  Some  key  technical  skills  were  not  represented  on  the  team  itself.  1 

2X 

3 

4 

5  9 

D7.  Team  meetings  were  sometimes  frustrating  and  non-productive.  1 

2 

3  X 

4 

5  9 

D8.  Professionals  were  split  across  too  many  different  tasks  &  teams.  1 

2 

3  X 

4 

5  9 

D9.  Project  results  did  not  take  advantage  of  the  team’s  best  ideas.  1 

2  X 

3 

4 

5  9 

DIO.  Key  members  continued  through  pre-production  planning 

and  testing.  1 

2 

3 

4  X 

5  9 

Dll.  There  was  often  uncertainty  about  the  future  of  project  funding.  1 

2 

3 

4 

5  X  9 

D12.  The  team  was  reluctant  to  share  concerns  with  gov’t  PM  .  1  X 

2 

3 

4 

5  9 

D13.  Management  project  reviews  were  constructive  &  helpful.  1 

2 

3 

4  X 

5  9 

D14.  Formal  reviews  were  conducted  at  key  decision  points.  1 

2 

3 

4 

5  X  9 

D15.  At  theprime  contractor,  the  project  was  a  management  priority.  1 

2 

3 

4 

5  X  9 

D16.  Usually  team  knew  right  awav  where  to  get  necessary  outside  help.l 

2 

3 

4  X 

5  9 

D17.  Project  had  a  visible  &  supportive  champion  in  the  Prime’s  management.  1 

2 

3 

4 

5  X  9 

D18.There  was  a  lot  of  contact  with  TRADOC*  during  the  project.  1 

2 

3 

4 

5  X  9 

D19.  The  gov’t  PM  was  reluctant  to  share  problems  with  Army  leaders  X  2  3  4  5  9 

•  By  TRADOC  here  and  elsewhere,  we  mean  Training  &  Doctrine  Command  and/or  other  appropriate  user  representatives. 

D20.  Who  besides  the  team  usually  attended  formal  reviews?  (Check  all  that  apply.) 

D20a.  Any  Prime  upper  management  (Director  or  VP  level)?  X  1.  Yes  _ 

D20b.  Any  Army  Program  management  representatives?  _X  1.  Yes  _ 

D20c.  Any  TRADOC  or  other  user  representatives?  _  X  1.  Yes  _ 

2.  No  _ 

2.  No  _ 
2.  No  _ 

8.  Not  appl.  _ 9.  DK 

_  8.  Not  appl.  _ 9.  DK 

_  8.  Not  appl.  _ 9.  DK 

Activity  ReDort  durina  Svstem  DeveloDment  Staae  of  Proiect 

How  often  did  team  members  do  the  followina  durina  DeveloDment? 

(If  you  feel  the  activity  is  Not  Applicable  to  your  project,  check  NA.)  Never 

FI.  Went  to  the  shop  floor  to  meet  about  related  production  processes.  _ 

Once  or  ! 

twice 

Several  Many  Don’t  know, 
times  times  Not  aool 

_  _X_  (_) 

F2.  Asked  for  supplier  comments  &  suggestions  on  design  choices. 

X 

(  ) 

F3.  Showed  &  discussed  physical  models  of  new  components  with  suppliers. 

_x_ 

_  (_) 

F4.  Needed  management  help  to  resolve  project  team  disagreements. 

X 

(  ) 

How  often  did  the  following  occur  during  Development? 

F5.  Did  TRADOC/other  user  organizations  show  strong  support? 

_x_  (_) 

F6.  Were  there  changes  in  key  TRADOC  or  other  user  personnel? 

X 

(  ) 

F7.  Were  there  changes  in  system  requirements  (e.g.,  threat)? 

X 

—  - 

— 

_  (_) 

SFfARF.D  dfstgn -production  acttvtttfs  during  Svstem  Development.  Here  only  count  ioint  meetines  or 

discussions  that  included  both  DESIGN  and  people  from  PRODUCTION 
and/or  from  the  PROGRAM  concerned  with  production  of  the  System, 
know, 

How  often  did  team  members  do  the  following  during  Development? 

F10.  Passed  around  physical  prototypes  during  joint  discussions. 

Once  or 

Never  twice 

Several  Many  Don’t 

times  times  Not  aool 

_  _x_  (_) 

FI  1.  Held  planning  meetings  that  included  both  design  &  production  people. 

X 

(  ) 

F12.  Explored  choices  together  with  computational  models  or  analytic  tools. 

_x_ 

_  (_) 
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The  Manufacturing  engineers  reviewed  prints  all  throughout  the  project,  but  they  didn’t  use  computational  models  or  analytic 
tools.  Computer  tools  didn’t  exist  at  the  time,  and  they  didn’t  have  any  manufacturing  analytical  tools  -  1970s  and  early 
1980s. 


F13.  Had  test  articles  or  pre-production  parts  to  discuss  and  examine  jointly. 

X 

(  ) 

SHARED  DESIGN-SUPPLIER  ACTIVITIES  during  Svstem  Development.  Now  onlv  count  ioint  meetings  or 
discussions  that  included  personnel  from  both  DESIGN  and  SUPPLIERS. 

How  often  did  team  members  do  the  following  during  Development? 

know, 

Once  or 

Several  Many  Don’t 

Never 

twice 

times  times 

Not  aool. 

F20.  Passed  around  physical  prototypes  during  joint  discussions.  X 

(  ) 

F21.  Held  planning  meetings  that  included  both  design  and  suppliers. 

X 

(_) 

F22.  Explored  choices  together  with  computational  models  or  analytic  tools.  _ X _ 

(_) 

F23.  Had  test  articles  or  pre-production  parts  to  discuss  and  examine  jointly. 

_x_ 

(_) 

Design  engineers  and  suppliers  wotked  closely  together.  This  was  a  very  unique  design  so  the  suppliers  were 
designing/tailoring  their  hardware  for  this  specific  job  in  many  cases. 


ACTIVITY  PHASING  BY  STAGES  OF  DEVELOPMENT  AND  TRANSITION 
WHEN  were  the  following  activities  carried  out  by  the  team?  For  example,  if  for  Wl.  Production  was  involved 
regularly  in  the  Selection/Planning  stage,  dropped  out,  and  then  came  back  in  late  in  the  Development  work  and  continued  to 
participate  after  that,  check  (/)  first,  fourth  and  fifth  columns. 

SP  D  TP 

I  Selection  I  Development  I  Transition  (DK/ 

Please  check  (/’ )  all  stages  when  the  activity  occurred.  V  Tarly  Middle  Later  W  (Never)  NA) 


Wl.  When  did  production  representatives  participate  regularly? 

X  X 

L  )(  ) 

W2.  When  did  team  members  meet  with  production  on  shop  floor? 

W3.  When  was  the  TRADOC  consulted  on  project  questions? 

W4.  When  was  there  change  in  key  TRADOC/user  representatives? 

The  TRADOC  Systems  Manager  and  other  military  personnel  change 
know  exactly  when  these  changes  occurred.  However,  I  don’t  think  tl 

W5.  When  did  TRADOC/other  users  show  strong  support? 

W6.  When  was  there  change  in  the  system  requirements? 

Relationship  &  Activities  between  Enaineerina  Desian  & 

X 

_ X 

X  X  X 

__x_  (__)(_) 

X  (_)(_) 

1  about  eve 
is  was  eve 

X 

y  three  years  as  I  re 
a  problem. 

XXX 

(  )(_X  ) 

nember,  but  I  don’t 

X  (_  )(_) 

Product 

on/Proaram 

_x _ ( _ )( _ ) 

Transition  (DK/ 

These  questions  are  different  because  they  focus  only 
on  ioint  meetinas  or  discussions  that  included  both  S 

DESIGN  personnel  and  people  from  PRODUCTION 
and/or  PROGRAM  people  concerned  with  production.  I 

W16.  When  did  the  team  &  technical  professionals  from  Production 
have  unscheduled  &  informal  ioint  conversations  about  the  nroiect? 

W17.  When  were  analytic  engineering  tools  used  iointlv  bv  Design 
and  Production  to  explore  options  together? 

5  D 

Selection 

TF 

Development 

r  ' 

r  Early  Middle  Later1 

X _ X  _X_ 

f  (Never)  NA) 

_ _  (_)  (_) 

(_)  (X  ) 

W18.  When  were  prototypes  and  parts  used  in  ioint  discussions? 

Relationship  &  Activities  between  Enqineerinq  Desiqn  & 

Supplie 

X 

rs 

X  (_)  <_) 

ransition  (DK/ 

Focus  onlv  on  ioint  meetinas  or  discussions  that  S! 

included  both  DESIGN  personnel  and  SUPPLIERS: 

W26.  When  did  the  team  &  technical  professionals  from  Suppliers  } 
have  unscheduled  &  informal  ioint  conversations  about  the  oroiect? 

W27.  When  were  analytic  engineering  tools  used  iointlv  bv  Design 
and  Suppliers  to  explore  options  together? 

3  D 

Selection 

TP 

Development 

r  i 

x 

Early  Middle  Later  ^ 

X  _x_x  _x_ 

^  (Never)  NA) 

_ _  (_)  (_) 

( _  )  (  x  ) 
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W28.  When  were  prototypes  and  parts  used  in  joint  discussions? 


_ X__X_  (_)  (_) 


Problem  Solving  and  Team  Effort 

Here  are  a  series  of  statements  about  problems  that  are  said  to  occur  with  technology  development.  For  each  statement,  we 
are  asking  you  to  make  two  separate  judgments  to  help  us  understand  what  problems  require  substantial  team  effort: 

O  First,  did  this  problem  ever  come  up  in  the  specific  project  being  reported  on?  If  “No”,  then  circle  the  “0”. 

©  If  “Yes,”  how  serious  was  the  impact  of  this  problem  on  the  process  of  the  project’s  work?  Here  we  are  concerned  with 
how  much  effort  in  attention,  tune  and  energy  did  the  project  have  to  spend  solving  or  compensating  for  this  problem. 


0  Did  this  problem  come  up  during  this  project? 

No. 

Yes.  The  problem  came  up. 

0  IF 

YES.  howmuch  nroiec 

effort 

had  to  be  scent  on 

this  oroblem? 

Very 

Very 

minor 

Minor 

Sigmf. 

Major 

major 

effort 

effort 

effort 

effort 

effort 

Bl.  It  was  harder  than  expected  to  take  the  risk  out  of  the  new  technology. 

0 

i 

2 

3 

4X 

5 

B2.  Cut-backs  in  project  resources  forced  changes/compromises. 

0 

IX 

2 

3 

4 

5 

B3.  Changes  in  company  strategies  and  goals  hurt  the  project. 

OX 

i 

2 

3 

4 

5 

B4.  A  critical  production  issue  was  uncovered  very  late  in  the  process. 

0 

i 

2X 

3 

4 

5 

B5.  Management  pressure  pushed  technology  prematurely  into  production. 

0 

i 

2X 

3 

4 

5 

B6.  There  was  a  lack  of  acceptance  standards  for  the  new  technology. 

0 

IX 

2 

3 

4 

5 

B7.  The  technology  was  hard  to  scale  up  from  lab  &  pilot  tests. 

0 

l 

2 

3X 

4 

5 

B8.  Testing,  quality  control  and/or  acceptance  took  longer  than  planned 

0 

i 

2 

3X 

4 

5 

B9.  Departments  at  the  prime  resisted  project  ideas  &  approaches. 

0 

i 

2X 

3 

4 

5 

BIO.  One  or  more  suppliers  did  not  meet  their  commitments. 

0 

l 

2 

3X 

4 

5 

Bl  1.  Army  Labs/Centers  resisted  project  ideas  or  approaches. 

X 

i 

2 

3 

4 

5 

B12.  Army  program  offices  resisted  project  ideas  or  approaches. 

0 

IX 

2 

3 

4 

5 

B13.  Threat  definition  or  other  requirements  changed  during  the  project. 

OX 

1 

2 

3 

4 

5 

Project  Outcomes 

01.  Project  Acceptance.  Was  the  SYSTEM  accepted  to  be  put  into  Production?  This  is  initial  acceptance,  not  whether  it 
actually  ended  up  in  production. 

_  1.  No,  the  System  was  abandoned.  _  8.  NA,  not  applicable 

_ 2.  No,  but  concept/technology  was  used  later.  _  9.  DK,  don’t  know/can’t  remember 

_X _ 3.  Yes,  the  System  was  accepted  for  production 

02.  After  the  SYSTEM  was  accepted  and  was  in  Transition  to  Production,  how  many  additional  changes  in  the  designs 
and  processes  were  later  required  before  the  System  was  taken  into  lull  production? 

_X _ 1.  Many  serious  changes  _ 2.  Significant  changes  _ 3.  Minor  changes  _ 4.  No  or  almost  no  changes 

_  7.  Did  not  reach  production,  was  not  implemented  _ 8.  Not  Applicable  _ 9.  Don’t  know 

There  was  a  large  amount  of  work.  TADS  pointing  angle  accuracy  was  a  big  problem.  They  had  to  work  on  getting  a  noise- 
free  FLIR.  And  they  needed  to  work  on  consistency  of  Line-of-Sight  (LOS)  Stabilization  -  they  made  repeated  changes  to 
meet  this  specification  requirement.  The  delivery  rate  of  10.  and  then  12  per  month  exacerbated  the  problem. 

03.  Did  the  SYSTEM  go  into  full  production? 

_  1.  No,  the  System  was  abandoned.  _ 8.  NA,  not  applicable 

_  2.  No,  but  concept/some  technology  was  used  later.  _  9.  DK,  don’t  know/can’t  remember 

_X _  3.  Yes,  the  System  was  put  into  fall  production. 
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04.  For  each  of  the  technologies  A,  B,  and  C  above,  to  what 

04a 

04b. 

04c. 

extent  was  each  used  in  the  System  as  it  was  produced: 

Technology  A 

Technology  B 

Technology  C 

No,  the  technology  was  not  used  in  the  System. 

i.  _ 

i._ 

i._ 

No,  but  the  technology  was  used  later(elsewhere) 

2. _ 

2. _ 

2. _ 

The  technology  was  used  but  not  to  extent  originally  planned. 

3. _ 

3. _ 

3. _ 

Yes,  the  technology  was  used  as  planned. 

4.  _X _ 

4.  _X_ 

4.X _ 

Don’t  know. 

9.  _ 

9.  _ 

9.  _ 

After  the  early  stages  of  development  LOS  stabilization  never  really  became  an  issue  any  more.  FLIR  acquisition  ranges  were 
met  in  the  later  stages  of  development  and  were  not  a  problem  in  the  production  hardware.  Boresight  performance 
continued  to  be  an  issue  into  the  early  stages  of  production.  The  cost  of  the  fixes  were  not  major  but  took  a  lot  of  time  to 
work  out.  All  three  technologies  were  essential  to  the  performance  of  the  TADS  and  thus  had  to  be  successfully  used  in  the 
final  system. 


05.  After  the  SYSTEM  reached  Transition  to  Production^  did  the  project  go  to  Production  as  quickly  as  it  should  have? 

_  1.  No  delay  _ X_  2.  One  to  six  months  delay  _ 3.  Seven  to  twelve  month  delay  _ 4.  Over  a  year  late 

_  5.  Did  not  reach  production,  was  not  implemented  _ 8.  Not  Applicable  _ 9.  Don’t  know 

There  were  some  schedule  changes  from  earlier  schedules  but  I  can’t  remember  all  of  the  reasons  or  exactly  how  much 
change  and  from  what  baseline  schedule  the  changes  occurred. 

06.  After  the  SYSTEM  was  actually  in  Production,  howmany  additional  changes  in  designs  and  processes  were  required? 

_ 1.  Many  serious  changes  _ 2.  Significant  changes  _X _ 3.  Minor  changes  _ 4.  No  or  almost  no  changes 

_  7.  System  did  not  reach  production  _ 8.  Not  Applicable  _ 9.  Don’t  know 

Again,  the  contractor  was  incentivised  to  make  reliability  improvement  changes  and  under  the  warranty  program  could  make 
changes  to  improve  reliability  and  thereby  save  the  contractor  (and  ultimately  the  government)  money.  Producability  changes 
were  also  made  mostly  because  of  parts  that  were  no  longer  available. 

07.  D  id  the  SY STEM  as  it  was  implemented  meet  the  program 's  cost  goals? 

_X _ 1.  The  results  met  or  exceeded  cost  goals  _ 7.  System  did  not  reach  production. 

_ 2.  The  results  came  close  to  achieving  cost  goals  _ 8.  Not  applicable. 

_ 3.  The  results  fell  far  short  of  achieving  cost  goals  _ 9.  Don’t  know. 

08.  Did  the  System  Development  program,  as  implemented,  come  in  on  budget? 

_ 1.  The  project  met  or  under-ran  budget.  9.  Don’t  know. 

_ 2.  The  project  slightly  exceeded  budget. 

_X _ 3.  The  project  significantly  exceeded  budget. 

As  stated  above  there  were  significant  overruns  to  the  development  contracts.  The  “Maturity  phase  contract  with  Martin 
Marietta  started  off  at  about  $45M  and  ended  up  at  about  twice  that.  However,  TADS/PNVS  was  not  a  separate  line  item  in 
the  budget  but  was  just  part  of  the  AH-64  budget  and  this  overran  was  covered  within  the  AH-64  budget. 

09.  Did  the  System  as  it  was  implemented  meet  the  project’s  technical  goals  and  functional  requirements? 

_ 1.  The  results  met  or  exceeded  technical  goals  7.  Did  not  reach  production,  was  not  implemented. 

X _ 2.  The  results  came  close  to  achieving  technical  goals  _ 8.  Not  applicable. 

_ 3.  The  results  fell  far  short  of  achieving  technical  goals  _ 9.  Don’t  know. 


09.  Did  the  System  have  problems  in  the  field  under  operational  conditions  in  Desert  Storm? 

_ 1.  Yes,  problems  in  the  field  significantly  limited  the  system’s  effectiveness. 

_ 2.  Yes,  problems  in  the  field  caused  minor  problems  in  the  system’s  effectiveness.  _ 9.  Don’t  know,  question 

_X_  3.  No,  the  system  was  deployed  and  encountered  no  noticeable  loss  of  effectiveness,  not  applicable  to  this  system. 
_ 4.  No,  the  system  was  deployed  and  exceeded  expectations  of  its  effectiveness. 

IF  YOU  CHECKED  “1”  or  “2”  to  question  09,  what  did  the  field  problems  result  from?  Check  all  that  apply. 

09a _ The  System  did  not  meet  its  requirements. 

09b. _ Requirements  did  not  reflect  the  field  environment.  09e. _ Other: _ 

09c.  _ The  System  was  not  deployed  in  its  intended  role.  (Please  explain.) _ 

09d. _ Personnel  not  adequately  trained/prepared  to  use  the  System. 

II.  Now  that  you  have  had  a  chance  to  think  about  the  project  and  provide  some  answers,  how  well  do  you  think  you  feel 
you  have  captured  the  details  of  the  project?  Are  you:  (Check  /  one.) 
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_X _ 1.  Very  confident  that  you  captured  the  project  well? 

_ 2.  Fairly  confident  you  understand  the  main  things  well,  but  not  as  confident  about  the  details? 

_ 3.  Not  confident  of  your  information  about  the  project,  so  we  should  only  use  your  answers  with  caution. 

Except  for  those  items  I  checked  as  “don’t  know”  and  taking  into  account  my  added  notes,  I  feel  confident  that  I’ve  given  a 
good  picture  of  what  happened. 
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12.  Finally,  what  was  the  most  difficult  problem  the  Project  Manager  faced,  how  was  the  problem  dealt  with,  and  what  was 
the  impact  of  the  problem  on  the  project  outcome? 

The  biggest  problem  was  the  cost  overruns  and  the  underlying  reasons  for  these  over-runs  in  development.  The  source  of 
these  cost  overruns  was  due  to  a  couple  of  factors.  The  primary  problem  was  probably  associated  with  the  acquisition 
approach.  After  the  fly-off  there  was  a  down-select  to  the  winning  contractor.  That  down-select  was  made  in  the  form  of  the 
contract  award  for  the  maturity  phase  of  the  development  program .  Each  contractor  submitted  a  proposal  for  the  m  aturity 
phase  as  part  of  the  down-select  competition.  It  certainly  was  not  in  the  contractors’  best  interest  at  that  stage  to  cost  the 
program  in  their  proposals  to  fully  cover  all  risk  areas.  First,  the  contractor  would  not  go  out  of  his  way  to  highlight  areas  of 
risk  that  the  government  team  had  not  identified  and  secondly,  for  those  areas  of  risk  that  were  identified  the  contractors  did 
not  want  to  indicate  the  full  cost  upside  of  the  risk  thereby  putting  themselves  at  a  substantial  competitive  disadvantage  for  the 
down-select  (both  by  proposing  a  higher  cost  than  their  competitor  and  by  highlighting  greater  technical  risk  with  their 
designs  that  required  greater  funding  in  the  maturity  phase  to  correct).  Having  said  all  of  this,  I  do  not  feel  that  the  acquisition 
approach  was  necessarily  the  wrong  one  because  the  approach  with  the  competitive  fly-off  and  a  subsequent  maturity  phase  is 
designed  to  significantly  reduce  the  risk  that  the  developed  systems  will  not  meet  performance  goals.  Since  the  TADS/PNVS 
was  the  number  one  riskiest  technology  in  the  AAH  program  this  was  an  appropriate  approach,  and  one  that  was  ultimately 
successful.  Having  said  that.  I  think  that  both  the  government  team  and  the  contractor  underestimated  the  effort  it  would  take 
to  implement  the  maturity  phase  design  changes,  meet  all  performance  goals,  develop  test  equipment,  and  transition  to 
production.  The  project  manager,  therefore,  had  to  deal  with  all  the  issues  that  came  up  because  there  was  more  effort 
required  to  get  the  job  done  in  the  required  time  frame  than  had  been  planned  for.  The  solutions  to  almost  all  problems 
resulted  in  increased  cost. 


Technology  Readiness  Level  Scale 

1.  Basic  principles  observed  and  reported.  Scientific  research  beginsto  be  translated  into  applied  research  and 
development  concepts.  There  have  been  paper  studies  of  technology’s  basic  properties. 

2.  Technology  concept  and/or  application  formulated.  Practical  applications  have  been  invented.  Application  is  specu¬ 
lative  and  there  is  no  proof  or  detailed  analysis  to  support  the  assumptions.  Examples  are  still  limited  to  paper  studies. 

3.  Analytical  &  experimental  critical  function  and/or  characteristic  proof  of  concept.  Analytical  and  laboratory 
studies  have  physically  validated  analytic  predictions  of  separate  elements  of  the  technology.  Examples  include 
components  that  are  not  yet  integrated  or  representative 

4.  Component  and/or  bread  board  validation  in  lab  environment.  Basic  technological  components  are  integrated  to 
establish  that  pieces  will  work  together,  e.g.,  integration  of  ad  hoc  parts  in  lab.  This  is  relatively  “low  fidelity”  compared 
to  the  eventual  system. 

5.  Components  and/or  bread  board  validation  in  relevant  environment.  Fidelity  of  breadboard  technology  is 
significantly  increased.  Basic  components  integrated  with  reasonably  realistic  supporting  elements  so  the  technology  can 
be  tested  in  a  simulated  environment.  Examples  include  “high  fidelity”  laboratory  integration  of  components. 

6.  System/sub  system  model  or  prototype  demonstrated  in  a  relevant  environment.  Representative  model  or  prototype 
system,  which  is  well  beyond  the  breadboard  tested  for  TRL  5,  tested  in  a  relevant  environment.  Represents  a  major  step 
up  in  a  technology’s  demonstrated  readiness.  Examples  include  testing  a  prototype  in  a  high  fidelity  laboratory 
environment  or  in  a  simulated  operational  environment. 

7.  System  prototype  demonstr  ated  in  an  operational  environment.  Prototype  near  or  at  planned  operational  system. 
Represents  a  major  step  up  from  TRL  6,  requiring  the  demonstration  of  an  actual  system  prototype  in  an  operational 
environment,  such  as  in  an  aircraft,  vehicle  or  space.  Examples  include  testing  the  prototype  in  a  test  bed  aircraft. 

8.  Actual  system  completed  and  qualified  in  test  and  demonstration.  Technology  proven  to  work  in  final  form  and 
under  expected  conditions.  In  almost  all  cases,  this  TRL  represents  the  end  of  true  system  development.  Examples  include 
developmental  test  and  evaluation  in  its  intended  weapon  system  to  determine  if  it  meets  design  specification. 

9.  Actual  system  proven  in  successful  operational  environment.  Actual  application  of  the  technology  in  its  final  form 
and  under  mission  conditions,  such  as  those  encountered  in  operational  test  and  evaluation.  In  almost  all  cases,  this  is  the 
end  of  the  last  “bug  fixing”  aspects  of  hue  system  development.  Examples  include  using  the  system  under  operational 
mission  conditions. 

Production  Readiness  Scale 

1.  The  subsystem  or  component  application  embodying  the  technology  is  produced  inside  the  lab  by  engineers,  scientists 
or  laboratory  technicians  to  demonstrate  principles  for  breadboard  validation  and  testing. 

2.  The  application  is  produced  outside  the  lab  with  tools  and  processes  used  for  producing  very  low  quantities. 
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3.  The  application  is  produced  in  low  quantities  with  tools  and  processes  planned  to  be  used  in  production  systems.  Testing 
procedures  for  components  and  subsystems  are  established. 

4.  The  system  involving  the  technology  application(s)  is  engineered  for  production.  All  components  are  identified, 
integration,  assembly  and  test  planning  is  complete. 

5.  Low  rate  production  has  been  run  using  the  production  processes  planned  for  full  rate  production,  complete  with 
validated  procedures  for  integration,  assembly  and  test  of  the  system. 
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INITIAL  DISTRIBUTION  LIST 


1 .  Defense  Technical  Infonnation  Center 
Ft.  Belvoir,  Virginia 

2.  Dudley  Knox  Library 
Naval  Postgraduate  School 
Monterey,  California 

3.  Fredrick  Reed 
USA  AMCOM 
Redstone  Arsenal,  AL 

4.  Dr.  Willie  J.  Fitzpatrick 
USA  AMCOM 
Redstone  Arsenal,  AL 

5.  Bill  Craig 
USA  AMCOM 
Redstone  Arsenal,  AL 

6.  Dr.  Richard  Rhoades 
University  of  Alabama  at  Huntsville 
Huntsville,  AL 

7.  Professor  David  F.  Matthews 
Naval  Postgraduate  School 
Monterey,  CA 

8.  Professor  Brad  R.  Naegle 
Naval  Postgraduate  School 
Monterey,  CA 

9.  Dr.  David  Lamm 

Naval  Postgraduate  School 
Monterey,  CA 
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